Oct 31 22:47:11.717672 ip-10-0-131-157 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 22:47:11.717685 ip-10-0-131-157 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 22:47:11.717694 ip-10-0-131-157 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 22:47:11.718019 ip-10-0-131-157 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 22:47:21.917382 ip-10-0-131-157 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 22:47:21.917399 ip-10-0-131-157 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot da1c4571b4cf4d1c9c982095b93d800a -- Oct 31 22:49:10.369304 ip-10-0-131-157 systemd[1]: Starting Kubernetes Kubelet... Oct 31 22:49:10.808885 ip-10-0-131-157 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:10.808885 ip-10-0-131-157 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 22:49:10.808885 ip-10-0-131-157 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:10.808885 ip-10-0-131-157 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816702 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816794 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816797 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816800 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816804 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816806 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816809 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816812 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.816802 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816815 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816819 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816822 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816825 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816828 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816830 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816833 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816835 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816837 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816840 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816843 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816845 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816848 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816850 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816853 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816855 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816858 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816861 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816863 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816866 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.817125 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816869 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816871 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816873 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816876 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816878 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816881 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816884 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816887 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816889 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816891 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816894 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816896 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816899 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816901 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816905 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816908 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816911 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816914 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816916 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816919 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.817582 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816921 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816924 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816927 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816930 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816932 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816935 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816938 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816941 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816944 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816946 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816948 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816951 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816953 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816956 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816959 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816961 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816964 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816966 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816969 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816972 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.818066 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816976 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816980 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816982 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.816998 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817001 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817004 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817008 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817010 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817015 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817018 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817020 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817023 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817026 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817030 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817035 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817038 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817041 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817044 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817046 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.818538 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817049 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817052 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817055 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817058 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817060 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817063 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817065 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817068 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817070 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817073 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817076 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817078 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817082 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817084 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817087 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817089 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817092 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817094 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817097 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.818980 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817893 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817901 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817906 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817910 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817913 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817916 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817919 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817921 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817924 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817927 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817930 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817933 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817936 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817938 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817941 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817943 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817946 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817948 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817951 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.819464 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817954 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817956 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817959 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817962 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817964 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817967 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817969 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817972 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817974 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817977 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817980 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.817983 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818003 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818008 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818012 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818017 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818021 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818024 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818026 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818029 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.819917 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818032 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818034 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818037 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818040 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818044 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818046 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818049 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818052 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818054 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818057 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818059 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818062 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818064 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818067 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818070 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818072 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818075 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818078 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818080 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818083 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.820411 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818086 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818089 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818091 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818093 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818096 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818099 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818104 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818107 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818111 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818115 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818118 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818121 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818123 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818126 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818128 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818131 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818133 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818136 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.820877 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818139 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818141 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818144 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818146 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818149 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818152 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818155 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818157 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818160 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818163 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818165 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818168 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818171 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818174 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818176 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818179 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818182 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818185 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818187 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.821316 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818190 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818193 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818196 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818199 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818202 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818204 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818207 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818209 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818212 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.818215 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819512 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819520 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819524 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819527 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819530 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819533 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819536 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819538 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819541 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819544 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.821761 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819546 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819549 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819552 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819554 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819557 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819560 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819563 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819566 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819568 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819571 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819573 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819576 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819579 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819583 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819587 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819590 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819592 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819595 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819598 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.822250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819600 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819602 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819605 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819607 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819610 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819612 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819615 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819618 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819621 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819623 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819625 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819628 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819631 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819633 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819635 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819638 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819641 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819644 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819646 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819649 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.822691 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819652 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819654 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819657 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819659 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819662 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819664 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819667 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819669 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819672 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819674 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819676 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819679 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819682 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819684 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819687 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819689 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819691 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819694 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819697 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.823170 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819700 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819702 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819705 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819707 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819710 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819713 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819715 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819718 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819720 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819725 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819728 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819730 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819733 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819737 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819741 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819745 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819747 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819750 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819752 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.823614 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819756 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819758 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819761 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819763 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819766 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819769 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819771 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819774 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819776 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819779 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819781 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819784 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819787 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819790 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819792 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819795 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819798 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819801 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.819804 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820793 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820804 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 22:49:10.824095 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820812 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820817 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820822 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820825 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820830 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820835 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820838 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820841 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820845 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820848 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820851 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820854 2582 flags.go:64] FLAG: --cgroup-root="" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820857 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820860 2582 flags.go:64] FLAG: --client-ca-file="" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820863 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820866 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820870 2582 flags.go:64] FLAG: --cluster-domain="" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820873 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820876 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820879 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820882 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820887 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820890 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 22:49:10.824589 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820893 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820896 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820899 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820903 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820906 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820909 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820912 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820917 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820920 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820923 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820925 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820928 2582 flags.go:64] FLAG: --enable-server="true" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820931 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820939 2582 flags.go:64] FLAG: --event-burst="100" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820942 2582 flags.go:64] FLAG: --event-qps="50" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820945 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820948 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820951 2582 flags.go:64] FLAG: --eviction-hard="" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820956 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820959 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820962 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820965 2582 flags.go:64] FLAG: --eviction-soft="" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820968 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820971 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820974 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 22:49:10.825134 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820978 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820981 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820984 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.820999 2582 flags.go:64] FLAG: --feature-gates="" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821004 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821007 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821011 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821014 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821017 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821020 2582 flags.go:64] FLAG: --help="false" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821024 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821028 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821032 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821035 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821038 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821041 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821044 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821048 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821050 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821053 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821058 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821061 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821064 2582 flags.go:64] FLAG: --kube-reserved="" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821067 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 22:49:10.825730 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821069 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821072 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821075 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821078 2582 flags.go:64] FLAG: --lock-file="" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821081 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821084 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821087 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821093 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821096 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821099 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821102 2582 flags.go:64] FLAG: --logging-format="text" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821105 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821109 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821112 2582 flags.go:64] FLAG: --manifest-url="" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821115 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821119 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821122 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821127 2582 flags.go:64] FLAG: --max-pods="110" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821130 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821133 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821136 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821139 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821142 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821145 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821148 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 22:49:10.826318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821156 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821159 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821163 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821167 2582 flags.go:64] FLAG: --pod-cidr="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821170 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821173 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821177 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821180 2582 flags.go:64] FLAG: --port="10250" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821183 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821186 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ff613f49a18024d7" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821189 2582 flags.go:64] FLAG: --qos-reserved="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821192 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821195 2582 flags.go:64] FLAG: --register-node="true" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821198 2582 flags.go:64] FLAG: --register-with-taints="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821202 2582 flags.go:64] FLAG: --registry-burst="10" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821204 2582 flags.go:64] FLAG: --registry-qps="5" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821207 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821210 2582 flags.go:64] FLAG: --reserved-memory="" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821214 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821217 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821220 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821223 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821226 2582 flags.go:64] FLAG: --runonce="false" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821229 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821232 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821235 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 31 22:49:10.826907 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821238 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821241 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821246 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821250 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821253 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821256 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821259 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821262 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821265 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821269 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821273 2582 flags.go:64] FLAG: --system-cgroups="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821275 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821281 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821284 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821287 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821291 2582 flags.go:64] FLAG: --tls-min-version="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821294 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821297 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821300 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821303 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821307 2582 flags.go:64] FLAG: --v="2" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821311 2582 flags.go:64] FLAG: --version="false" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821316 2582 flags.go:64] FLAG: --vmodule="" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821320 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821324 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 22:49:10.827567 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821462 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821465 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821469 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821472 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821475 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821477 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821480 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821483 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821486 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821489 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821493 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821496 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821499 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821502 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821505 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821507 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821510 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821513 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821516 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821518 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.828190 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821521 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821524 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821526 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821529 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821531 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821534 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821537 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821540 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821542 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821545 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821547 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821550 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821553 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821555 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821558 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821561 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821564 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821567 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821569 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821572 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.828677 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821574 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821578 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821581 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821587 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821590 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821592 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821595 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821598 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821601 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821603 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821606 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821609 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821611 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821614 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821616 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821619 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821622 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821625 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821627 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.829165 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821630 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821632 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821635 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821638 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821640 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821643 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821645 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821648 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821651 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821654 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821656 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821659 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821662 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821664 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821668 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821671 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821674 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821677 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821680 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821683 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.829643 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821685 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821688 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821690 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821693 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821696 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821700 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821703 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821706 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821708 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821711 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821714 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821716 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821719 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821721 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821724 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821726 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821728 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821732 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821736 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.830129 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821739 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821742 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821745 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821747 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821750 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821753 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821755 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821759 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.821786 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821824 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821830 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821833 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821836 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821839 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821843 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821846 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.830575 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821850 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821853 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821857 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821859 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821863 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821866 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821868 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821871 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821873 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821876 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821879 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821883 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821887 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821890 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821895 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821898 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821900 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821902 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.830954 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821905 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821908 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821910 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821914 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821916 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821920 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821923 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821926 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821929 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821932 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821934 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821939 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821942 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821944 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821947 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821950 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821952 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821954 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821957 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821959 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.831477 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821962 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821964 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821967 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821969 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821972 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821975 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821977 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821980 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821982 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.821985 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822002 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822006 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822010 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822015 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822018 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822020 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822023 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822028 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822031 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822034 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.831949 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822036 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822039 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822042 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822047 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822049 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822052 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822055 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822057 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822060 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822063 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822066 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822069 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822071 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822074 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822077 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822079 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822082 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822084 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822087 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.832434 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822090 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822093 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822095 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822098 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822100 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822103 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822105 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822108 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822110 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822113 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822117 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822120 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822123 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822126 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822128 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822131 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822136 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822138 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822142 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822145 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.832919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.822147 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.829752 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.829770 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.829787 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.829792 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829850 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829857 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829860 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829863 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829866 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829868 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829871 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829874 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829877 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829881 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829883 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829887 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829889 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829892 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829895 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.833401 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829897 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829900 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829903 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829905 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829908 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829910 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829913 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829916 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829919 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829921 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829924 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829926 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829929 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829932 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829936 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829939 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829941 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829944 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829947 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829949 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.833919 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829952 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829955 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829957 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829960 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829963 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829965 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829968 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829970 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829973 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829975 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829979 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.829984 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830000 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830004 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830007 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830009 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830012 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830015 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830018 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.834438 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830021 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830023 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830026 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830029 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830032 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830034 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830037 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830040 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830043 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830046 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830050 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830052 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830055 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830057 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830060 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830063 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830065 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830068 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830070 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.835837 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830073 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830076 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830078 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830082 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830086 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830089 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830091 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830094 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830096 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830099 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830101 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830105 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830107 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830110 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830112 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830115 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830118 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830120 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830123 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.836303 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830125 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830129 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830131 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830135 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830138 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830141 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830143 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830146 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830148 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830151 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830154 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830156 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830159 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830161 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.830190 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830353 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.837035 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830359 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830363 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830367 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830369 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830372 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830375 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830378 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830380 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830384 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830386 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830389 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830391 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830394 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830397 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830399 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830402 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830405 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830407 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830410 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.837676 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830412 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830416 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830419 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830421 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830424 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830427 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830429 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830432 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830434 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830438 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830443 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830446 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830449 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830452 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830454 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830457 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830459 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830462 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830465 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.838153 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830467 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830470 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830473 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830476 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830479 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830481 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830483 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830486 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830489 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830492 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830494 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830497 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830499 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830502 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830504 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830508 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830510 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830513 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830515 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830518 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.838713 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830520 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830523 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830526 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830528 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830531 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830533 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830536 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830538 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830541 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830543 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830546 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830548 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830551 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830553 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830556 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830558 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830561 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830564 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830566 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.839250 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830569 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830572 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830574 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830577 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830580 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830583 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830586 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830588 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830591 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830597 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830600 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830602 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830605 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830607 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830610 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830613 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830615 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830617 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830620 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830623 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.839716 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830625 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830628 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830631 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830633 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830636 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830638 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830641 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830644 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.830666 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830698 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830702 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830705 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830707 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830710 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830713 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830715 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:10.840463 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830718 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830721 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830723 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830726 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830728 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830731 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830734 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830736 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830739 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830742 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830744 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830747 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830751 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830753 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830756 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830759 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830761 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830764 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830767 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830769 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:10.841104 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830773 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830776 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830779 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830782 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830785 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830788 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830790 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830793 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830795 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830798 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830800 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830803 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830805 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830808 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830810 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830813 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830815 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830818 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830820 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:10.841668 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830823 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830826 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830829 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830831 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830833 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830836 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830838 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830841 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830843 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830846 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830851 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830855 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830858 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830861 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830863 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830866 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830869 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830871 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830874 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:10.842150 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830879 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830881 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830884 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830887 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830889 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830892 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830895 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830897 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830900 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830902 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830905 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830908 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830910 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830913 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830916 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830919 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830921 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830924 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830926 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:10.842597 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830929 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830931 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830934 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830937 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830939 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830942 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830945 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830947 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830949 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830952 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830955 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830958 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830960 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830965 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830967 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830970 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830973 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830975 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830978 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830981 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:10.843374 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.830983 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:10.831011 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.831069 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.833451 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.834498 2582 server.go:1014] "Starting client certificate rotation" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.834606 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.834656 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.836761 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:49:10.843921 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.837867 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:49:10.850021 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.850004 2582 log.go:25] "Validated CRI v1 runtime API" Oct 31 22:49:10.855601 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.855583 2582 log.go:25] "Validated CRI v1 image API" Oct 31 22:49:10.856544 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.856519 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 22:49:10.860101 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.860081 2582 fs.go:135] Filesystem UUIDs: map[12d4ebba-a539-48c7-90c6-c7dc3ea13125:/dev/nvme0n1p3 43995d6f-7e30-4c43-ad9a-c5467c27c251:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 22:49:10.860155 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.860101 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 22:49:10.866284 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.866168 2582 manager.go:217] Machine: {Timestamp:2026-10-31 22:49:10.86484189 +0000 UTC m=+0.381696662 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113780 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db56fe036d80fd3eb8c89b929d11c SystemUUID:ec2db56f-e036-d80f-d3eb-8c89b929d11c BootID:da1c4571-b4cf-4d1c-9c98-2095b93d800a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:08:e9:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:08:e9:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:48:97:8f:54:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 22:49:10.866284 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.866282 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 22:49:10.866423 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.866411 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 22:49:10.867702 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867677 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 22:49:10.867835 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867705 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-157.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 22:49:10.867877 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867837 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 22:49:10.867877 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867846 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 22:49:10.867933 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867881 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:49:10.867933 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.867902 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:49:10.869103 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.869090 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 22:49:10.869412 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.869395 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 22:49:10.873684 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873673 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 22:49:10.873806 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873797 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 22:49:10.873837 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873832 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 31 22:49:10.873867 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873844 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 22:49:10.873867 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873856 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 22:49:10.873867 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873866 2582 kubelet.go:405] "Adding apiserver pod source" Oct 31 22:49:10.873972 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.873874 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 22:49:10.874239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.874214 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:49:10.876661 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.876645 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 22:49:10.884186 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.884173 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:49:10.885112 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885096 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 22:49:10.885155 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885131 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 22:49:10.885282 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885273 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 22:49:10.885310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885288 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 22:49:10.885310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885295 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 22:49:10.885310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885301 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 22:49:10.885310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885307 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 22:49:10.885414 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.885314 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 22:49:10.886028 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886017 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 22:49:10.886057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886038 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 22:49:10.886057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886046 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 22:49:10.886057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886051 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 22:49:10.886182 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886066 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 22:49:10.886182 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886075 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 22:49:10.886822 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886812 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 22:49:10.886822 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.886822 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 22:49:10.888785 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.888768 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s4xwb" Oct 31 22:49:10.892472 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.892455 2582 server.go:1263] "Started kubelet" Oct 31 22:49:10.892574 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.892516 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 22:49:10.892646 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.892602 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 22:49:10.892704 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.892671 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 22:49:10.893300 ip-10-0-131-157 systemd[1]: Started Kubernetes Kubelet. Oct 31 22:49:10.894214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.894197 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 22:49:10.894608 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.894520 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-157.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 22:49:10.894608 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.894562 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 31 22:49:10.895459 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.895302 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-157.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 22:49:10.895459 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.895300 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 22:49:10.898194 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.898177 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 22:49:10.901634 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.901619 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s4xwb" Oct 31 22:49:10.903244 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.903223 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 22:49:10.904433 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.904418 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 22:49:10.904554 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.904540 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:10.905083 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.903873 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-157.us-east-2.compute.internal.18e3bee1293db43b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-157.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-157.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 22:49:10.892418107 +0000 UTC m=+0.409272872,LastTimestamp:2026-10-31 22:49:10.892418107 +0000 UTC m=+0.409272872,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-157.us-east-2.compute.internal,}" Oct 31 22:49:10.905083 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905078 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 22:49:10.905214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905081 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 22:49:10.905214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905102 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 22:49:10.905214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905170 2582 factory.go:55] Registering systemd factory Oct 31 22:49:10.905214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905187 2582 factory.go:223] Registration of the systemd container factory successfully Oct 31 22:49:10.905340 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905217 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 31 22:49:10.905340 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905228 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 31 22:49:10.905456 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905406 2582 factory.go:153] Registering CRI-O factory Oct 31 22:49:10.905456 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905418 2582 factory.go:223] Registration of the crio container factory successfully Oct 31 22:49:10.905543 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905483 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 22:49:10.905543 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905504 2582 factory.go:103] Registering Raw factory Oct 31 22:49:10.905543 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.905522 2582 manager.go:1199] Started watching for new ooms in manager Oct 31 22:49:10.906715 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.906699 2582 manager.go:319] Starting recovery of all containers Oct 31 22:49:10.917876 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.917852 2582 manager.go:324] Recovery completed Oct 31 22:49:10.922370 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.922324 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:10.922537 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.922523 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:10.924259 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924242 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:10.924349 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924269 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:10.924349 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924288 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:10.924871 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924855 2582 cpu_manager.go:226] "Starting" policy="none" Oct 31 22:49:10.924871 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924863 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:10.924871 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924868 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 22:49:10.925048 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.924885 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 22:49:10.925532 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.925515 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:10.925599 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.925547 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:10.925599 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.925561 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:10.927028 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.927015 2582 policy_none.go:50] "Start" Oct 31 22:49:10.927068 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.927032 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 22:49:10.927068 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.927043 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 22:49:10.929080 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.929067 2582 policy_none.go:44] "Start" Oct 31 22:49:10.929540 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.929520 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-157.us-east-2.compute.internal\" not found" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:10.964686 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.964654 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 22:49:10.967156 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967139 2582 manager.go:341] "Starting Device Plugin manager" Oct 31 22:49:10.967244 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.967171 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 22:49:10.967244 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967182 2582 server.go:87] "Starting device plugin registration server" Oct 31 22:49:10.967464 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967451 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 22:49:10.967519 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967466 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 22:49:10.967583 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967571 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 22:49:10.967678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967664 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 22:49:10.967678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.967676 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 22:49:10.968215 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.968196 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 22:49:10.968323 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:10.968241 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:10.983292 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:10.983269 2582 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2614 into the system container \"/system.slice\": " Oct 31 22:49:11.067659 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.067566 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:11.068553 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.068534 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:11.068671 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.068561 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:11.068671 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.068574 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:11.068671 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.068606 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.092503 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.077397 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.092503 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.077419 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-157.us-east-2.compute.internal\": node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.092503 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.091914 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 22:49:11.094701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.093139 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 22:49:11.094701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.093162 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 22:49:11.094701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.093182 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 22:49:11.094701 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.093217 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 22:49:11.094898 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.094878 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.098514 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.098497 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:11.194163 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.194124 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal"] Oct 31 22:49:11.194291 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.194216 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:11.194982 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.194956 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.195269 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.195255 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:11.195326 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.195282 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:11.195326 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.195293 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:11.197522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.197509 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:11.197696 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.197661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.197696 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.197691 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:11.198262 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198246 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:11.198354 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198282 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:11.198354 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198298 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:11.198354 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198300 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:11.198354 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198319 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:11.198354 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.198328 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:11.200388 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.200373 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.200520 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.200398 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:11.201098 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.201081 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:11.201194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.201104 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:11.201194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.201115 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:11.206551 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.206528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.206637 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.206559 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.206637 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.206584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.229783 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.229765 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.232537 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.232517 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.295875 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.295843 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.307483 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.307628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.307628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.307628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307546 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.307628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307551 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.307628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.307609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.396974 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.396899 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.497481 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.497452 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.534908 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.534883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.537046 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.537030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.598420 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.598250 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.700643 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.700545 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:49:11.798402 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.798373 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:11.804789 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.804766 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.819484 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.819459 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:49:11.821734 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.821715 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:49:11.829719 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.829699 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:49:11.834149 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.834135 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 22:49:11.834258 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.834239 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 22:49:11.834317 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.834285 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 22:49:11.834317 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.834303 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 22:49:11.874498 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.874454 2582 apiserver.go:52] "Watching apiserver" Oct 31 22:49:11.901375 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.901345 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 22:49:11.903448 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.901996 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-nvttn","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal","openshift-dns/node-resolver-wzqjr","openshift-network-operator/iptables-alerter-wlwnn","openshift-ovn-kubernetes/ovnkube-node-8hd6q","kube-system/konnectivity-agent-zsmmj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-v5ftn","openshift-image-registry/node-ca-44szm","openshift-multus/multus-rdsnf","kube-system/global-pull-secret-syncer-qqjd2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk","openshift-cluster-node-tuning-operator/tuned-xzjmm","openshift-multus/multus-additional-cni-plugins-ctfvq"] Oct 31 22:49:11.903448 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.903367 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 22:49:11.904224 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.904174 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:44:11 +0000 UTC" deadline="2028-07-05 12:08:05.744138881 +0000 UTC" Oct 31 22:49:11.904301 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.904225 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14701h18m53.839917993s" Oct 31 22:49:11.904890 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.904874 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.907160 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.907140 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.909289 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909275 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:11.909758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909738 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\"" Oct 31 22:49:11.909844 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909837 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 22:49:11.909917 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909899 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 22:49:11.909958 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909940 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 22:49:11.910025 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.909996 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 22:49:11.911638 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:11.911784 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdtc\" (UniqueName: \"kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.911872 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.911872 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.911872 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912015 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912015 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.911964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912015 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912086 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912265 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912265 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912265 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912265 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912265 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912269 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912482 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rgk\" (UniqueName: \"kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912636 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.912711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.912675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:11.913949 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.913932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:11.916218 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.916191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:11.918460 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.918437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:11.920895 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.920879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:11.921555 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.921533 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 22:49:11.923309 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.923294 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:11.923392 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.923366 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:11.925641 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.925625 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:11.925706 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.925678 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:11.928650 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.928631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:11.928729 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:11.928688 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:11.931101 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.931087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:11.953793 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.953740 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-66ppd" Oct 31 22:49:11.964968 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:11.964944 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-66ppd" Oct 31 22:49:12.005998 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.005978 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 22:49:12.013443 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013558 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013606 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013688 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.013781 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47md\" (UniqueName: \"kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwqt\" (UniqueName: \"kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013909 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.013996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014035 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014031 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014108 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014142 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rgk\" (UniqueName: \"kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgqd\" (UniqueName: \"kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014394 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdtc\" (UniqueName: \"kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014504 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014532 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014548 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvb2\" (UniqueName: \"kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.014970 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.014975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015286 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015314 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.015821 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015418 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015447 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015498 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015562 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015528 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015680 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzz45\" (UniqueName: \"kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfxx2\" (UniqueName: \"kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.016501 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015878 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.015974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016040 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016163 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzddt\" (UniqueName: \"kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.017189 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016423 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016541 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016546 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016595 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.016632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.017483 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.017703 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.017503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.019092 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.019073 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 22:49:12.019223 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.019077 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021182 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021229 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021187 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021251 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021219 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021216 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021264 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021255 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021291 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021282 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021281 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 22:49:12.021321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021313 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\"" Oct 31 22:49:12.022042 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021480 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.022042 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021499 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.022042 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021528 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 22:49:12.022042 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021536 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\"" Oct 31 22:49:12.022042 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.021551 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 22:49:12.023717 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023686 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023740 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023740 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023770 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023802 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023814 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023817 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 22:49:12.023843 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.023831 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\"" Oct 31 22:49:12.024194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.024100 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\"" Oct 31 22:49:12.024194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.024122 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 22:49:12.024194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.024123 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\"" Oct 31 22:49:12.024194 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.024170 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 22:49:12.028585 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.028557 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdtc\" (UniqueName: \"kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.029082 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.029060 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rgk\" (UniqueName: \"kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.117320 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.117320 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgqd\" (UniqueName: \"kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvb2\" (UniqueName: \"kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.117455 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117498 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117508 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.117550 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:12.617525863 +0000 UTC m=+2.134380636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:12.117631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117596 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117717 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117767 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117884 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.117968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.118239 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118032 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzz45\" (UniqueName: \"kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118147 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfxx2\" (UniqueName: \"kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118316 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzddt\" (UniqueName: \"kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.119229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118524 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47md\" (UniqueName: \"kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwqt\" (UniqueName: \"kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120225 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118781 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118819 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118849 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118875 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118915 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.118927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119163 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119255 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119353 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119419 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119458 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119490 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119504 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119537 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.120758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119547 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119552 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119580 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119636 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119695 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.119716 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.119761 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:12.619745868 +0000 UTC m=+2.136600623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119804 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119807 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.119918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120046 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120431 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120457 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120470 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.121277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.120827 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.132231 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.132209 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgqd\" (UniqueName: \"kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.132339 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.132264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvb2\" (UniqueName: \"kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.133629 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.133606 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:12.133629 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.133628 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:12.133766 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.133657 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:12.133766 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.133741 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:12.633727061 +0000 UTC m=+2.150581829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:12.135844 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.135820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfxx2\" (UniqueName: \"kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.135844 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.135828 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzz45\" (UniqueName: \"kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.135960 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.135821 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzddt\" (UniqueName: \"kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.141711 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.141687 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwqt\" (UniqueName: \"kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.141832 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.141818 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47md\" (UniqueName: \"kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.196420 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.196382 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a68def89a8e2f4b1d88fecf85bfdb5e.slice/crio-0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca WatchSource:0}: Error finding container 0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca: Status 404 returned error can't find the container with id 0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca Oct 31 22:49:12.196936 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.196906 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de75e4a93b82a0133be1d6ed9e0a51.slice/crio-187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506 WatchSource:0}: Error finding container 187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506: Status 404 returned error can't find the container with id 187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506 Oct 31 22:49:12.202183 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.202163 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:49:12.228667 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.228638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdsnf" Oct 31 22:49:12.238723 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.238696 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b13f57_54cc_4499_a9a5_44eec96c4488.slice/crio-2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b WatchSource:0}: Error finding container 2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b: Status 404 returned error can't find the container with id 2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b Oct 31 22:49:12.247439 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.247420 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:49:12.253776 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.253751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:49:12.257612 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.257580 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91b6ba2_b532_47ef_b1de_ea7a018f2355.slice/crio-3946571f738eaad1ce59149716acf76f8aea280e6b276fed1f874ca9c7aa623b WatchSource:0}: Error finding container 3946571f738eaad1ce59149716acf76f8aea280e6b276fed1f874ca9c7aa623b: Status 404 returned error can't find the container with id 3946571f738eaad1ce59149716acf76f8aea280e6b276fed1f874ca9c7aa623b Oct 31 22:49:12.262884 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.262857 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ef2f42_14fa_4d62_be14_ef9fe9fd92f5.slice/crio-89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c WatchSource:0}: Error finding container 89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c: Status 404 returned error can't find the container with id 89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c Oct 31 22:49:12.265552 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.265529 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:12.269778 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.269759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:49:12.279240 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.279211 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39248804_861a_4442_a717_66a2467e2c72.slice/crio-46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217 WatchSource:0}: Error finding container 46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217: Status 404 returned error can't find the container with id 46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217 Oct 31 22:49:12.287701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.287677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:12.293319 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.293300 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:12.298792 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.298762 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fe7d02_92f2_41fd_9122_abdfdc159006.slice/crio-fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49 WatchSource:0}: Error finding container fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49: Status 404 returned error can't find the container with id fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49 Oct 31 22:49:12.302735 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.302711 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b2c63f_8550_4152_9759_d522b6b4e520.slice/crio-ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea WatchSource:0}: Error finding container ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea: Status 404 returned error can't find the container with id ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea Oct 31 22:49:12.311385 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.311360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44szm" Oct 31 22:49:12.321199 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.321175 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4544eebe_acaf_4a13_bd4e_c3df8a8fbc0d.slice/crio-903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74 WatchSource:0}: Error finding container 903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74: Status 404 returned error can't find the container with id 903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74 Oct 31 22:49:12.353182 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.353140 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:49:12.355368 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.355329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:49:12.365740 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.365712 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b6f6c7_3878_491d_8a05_131c56390f37.slice/crio-6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b WatchSource:0}: Error finding container 6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b: Status 404 returned error can't find the container with id 6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b Oct 31 22:49:12.367313 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:12.367277 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac9bdf_c637_46e4_9fac_53c43f3fd161.slice/crio-6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66 WatchSource:0}: Error finding container 6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66: Status 404 returned error can't find the container with id 6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66 Oct 31 22:49:12.369521 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.369497 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-m56nk_openshift-cluster-csi-drivers(48ac9bdf-c637-46e4-9fac-53c43f3fd161): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 22:49:12.370498 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.370477 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-m56nk_openshift-cluster-csi-drivers(48ac9bdf-c637-46e4-9fac-53c43f3fd161): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 22:49:12.372714 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.372680 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-m56nk_openshift-cluster-csi-drivers(48ac9bdf-c637-46e4-9fac-53c43f3fd161): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 22:49:12.372785 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.372743 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" podUID="48ac9bdf-c637-46e4-9fac-53c43f3fd161" Oct 31 22:49:12.623376 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.623281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:12.623376 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.623330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:12.623586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.623440 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:12.623586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.623440 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:12.623586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.623507 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:13.623487895 +0000 UTC m=+3.140342656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:12.623586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.623522 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:13.62351509 +0000 UTC m=+3.140369844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:12.724103 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.724064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:12.724271 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.724202 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:12.724271 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.724223 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:12.724271 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.724233 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:12.724405 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:12.724278 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:13.724264863 +0000 UTC m=+3.241119636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:12.968524 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.968358 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:12 +0000 UTC" deadline="2028-03-25 22:49:09.791923633 +0000 UTC" Oct 31 22:49:12.968524 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.968423 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12263h59m56.823505303s" Oct 31 22:49:12.975090 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.974553 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:12.986689 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:12.986638 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:13.095090 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.095045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:13.095285 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.095191 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:13.120874 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.120789 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44szm" event={"ID":"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d","Type":"ContainerStarted","Data":"903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74"} Oct 31 22:49:13.129632 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.129588 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zsmmj" event={"ID":"59b2c63f-8550-4152-9759-d522b6b4e520","Type":"ContainerStarted","Data":"ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea"} Oct 31 22:49:13.147682 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.147631 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49"} Oct 31 22:49:13.153832 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.153789 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wlwnn" event={"ID":"39248804-861a-4442-a717-66a2467e2c72","Type":"ContainerStarted","Data":"46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217"} Oct 31 22:49:13.187857 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.187812 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wzqjr" event={"ID":"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5","Type":"ContainerStarted","Data":"89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c"} Oct 31 22:49:13.211208 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.211163 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" event={"ID":"f91b6ba2-b532-47ef-b1de-ea7a018f2355","Type":"ContainerStarted","Data":"3946571f738eaad1ce59149716acf76f8aea280e6b276fed1f874ca9c7aa623b"} Oct 31 22:49:13.217034 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.216980 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdsnf" event={"ID":"f6b13f57-54cc-4499-a9a5-44eec96c4488","Type":"ContainerStarted","Data":"2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b"} Oct 31 22:49:13.223598 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.221907 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca"} Oct 31 22:49:13.229406 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.229371 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506"} Oct 31 22:49:13.238883 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.238845 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" event={"ID":"48ac9bdf-c637-46e4-9fac-53c43f3fd161","Type":"ContainerStarted","Data":"6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66"} Oct 31 22:49:13.250261 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.250215 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:007ba215236827a40e1b33f3799d289065305ae886c05e51214d74d0ff24d62d\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a5e96bc702bc529b424e6c80f95d7d4fc4fe92f83df4a77c3b47eb3a30fe729\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4db2cbbaecb77e6b3a4e8dd7ee7a4d4fa9ab69787e4330711a3abb1277af2ac\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" podUID="48ac9bdf-c637-46e4-9fac-53c43f3fd161" Oct 31 22:49:13.253143 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.253104 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerStarted","Data":"6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b"} Oct 31 22:49:13.632455 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.632364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:13.632635 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.632470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:13.632635 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.632523 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:13.632635 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.632592 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:13.632635 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.632618 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:15.63259661 +0000 UTC m=+5.149451371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:13.632860 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.632647 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:15.632630335 +0000 UTC m=+5.149485093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:13.733449 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.733407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:13.733648 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.733581 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:13.733648 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.733602 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:13.733648 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.733616 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:13.733806 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:13.733685 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:15.733667259 +0000 UTC m=+5.250522029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:13.969027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.968931 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:12 +0000 UTC" deadline="2028-06-13 12:50:15.651981505 +0000 UTC" Oct 31 22:49:13.969027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:13.968977 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14174h1m1.683009571s" Oct 31 22:49:14.018541 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:14.018510 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:14.095038 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:14.093766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:14.095038 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:14.093876 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:14.095038 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:14.094135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:14.095038 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:14.094248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:15.097613 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:15.097578 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:15.098140 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.097770 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:15.654077 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:15.653643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:15.654077 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:15.653708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:15.654077 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.653824 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:15.654077 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.653907 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.653883193 +0000 UTC m=+9.170737951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:15.654431 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.653830 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:15.654431 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.654346 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.654328749 +0000 UTC m=+9.171183504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:15.754845 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:15.754795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:15.755101 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.755014 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:15.755101 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.755050 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:15.755101 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.755065 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:15.755284 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:15.755125 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:19.755106211 +0000 UTC m=+9.271960970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:16.093606 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:16.093576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:16.093777 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:16.093711 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:16.094078 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:16.094029 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:16.094205 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:16.094180 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:16.285789 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:16.285749 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="487865b2eccb4a82bc27086d98d762941bd26843ce5c747dfdba4978e35b129a" exitCode=0 Oct 31 22:49:16.286336 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:16.285821 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"487865b2eccb4a82bc27086d98d762941bd26843ce5c747dfdba4978e35b129a"} Oct 31 22:49:17.095918 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.095145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:17.095918 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:17.095541 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:17.299321 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.298446 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zsmmj" event={"ID":"59b2c63f-8550-4152-9759-d522b6b4e520","Type":"ContainerStarted","Data":"77652153fc5a9620b4cabc5ebacc51da61a4acc06b48938c09f6eb4398acd5c8"} Oct 31 22:49:17.309329 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.309292 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a68def89a8e2f4b1d88fecf85bfdb5e" containerID="bc707c338344f8d3decfb90232747a175724a8d1800f58b693eb6f3fedfd5880" exitCode=0 Oct 31 22:49:17.309516 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.309402 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerDied","Data":"bc707c338344f8d3decfb90232747a175724a8d1800f58b693eb6f3fedfd5880"} Oct 31 22:49:17.317228 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.317193 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"6d4ec94d4947568ba04ce8da2a0fa5050782627fc7eccfa40a3c733897aec838"} Oct 31 22:49:17.340645 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.340544 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zsmmj" podStartSLOduration=1.774814316 podStartE2EDuration="6.340523141s" totalImagesPullingTime="4.565708825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:17.320316979 +0000 UTC m=+6.837171760" watchObservedRunningTime="2026-10-31 22:49:17.340523141 +0000 UTC m=+6.857377933" Oct 31 22:49:17.360219 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:17.360147 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" podStartSLOduration=5.36012611 podStartE2EDuration="5.36012611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:17.359057173 +0000 UTC m=+6.875911950" watchObservedRunningTime="2026-10-31 22:49:17.36012611 +0000 UTC m=+6.876980890" Oct 31 22:49:18.094666 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.094346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:18.094666 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:18.094477 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:18.094666 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.094551 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:18.094666 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:18.094637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:18.323148 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.322286 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"6e81279d3f58a67f979dda36c603d642af1d0dc7fa186d1a7b6c63be97d7c7b5"} Oct 31 22:49:18.327183 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.326276 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44szm" event={"ID":"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d","Type":"ContainerStarted","Data":"107178eec59e91070243143045a7bd63833e2d0672259c7487220f6b4f6aaf78"} Oct 31 22:49:18.345014 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.344143 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" podStartSLOduration=6.344123529 podStartE2EDuration="6.344123529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:18.343342413 +0000 UTC m=+7.860197192" watchObservedRunningTime="2026-10-31 22:49:18.344123529 +0000 UTC m=+7.860978312" Oct 31 22:49:18.364170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:18.364091 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-44szm" podStartSLOduration=1.799634895 podStartE2EDuration="7.364057793s" totalImagesPullingTime="5.564422898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:18.363929828 +0000 UTC m=+7.880784609" watchObservedRunningTime="2026-10-31 22:49:18.364057793 +0000 UTC m=+7.880912577" Oct 31 22:49:19.095202 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:19.095167 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:19.095400 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.095330 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:19.690582 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:19.690536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:19.691149 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:19.690634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:19.691149 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.690682 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:19.691149 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.690728 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:19.691149 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.690749 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:27.690732558 +0000 UTC m=+17.207587315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:19.691149 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.690766 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:27.690755662 +0000 UTC m=+17.207610442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:19.791330 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:19.791244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:19.791516 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.791465 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:19.791516 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.791488 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:19.791516 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.791502 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:19.791698 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:19.791558 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:27.791542621 +0000 UTC m=+17.308397389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:20.094076 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:20.093960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:20.094232 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:20.094122 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:20.094516 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:20.094495 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:20.094635 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:20.094614 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:20.204539 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:20.204503 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:20.205240 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:20.205220 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:21.095215 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.095173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:21.095678 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:21.095362 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:21.334118 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.334031 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wzqjr" event={"ID":"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5","Type":"ContainerStarted","Data":"af2e83d002fed7da7b07f90a44c74b72eba0aa954a35b367f892fb9eafad37f0"} Oct 31 22:49:21.335818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.335788 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdsnf" event={"ID":"f6b13f57-54cc-4499-a9a5-44eec96c4488","Type":"ContainerStarted","Data":"89a68f5ada811e47cf640ce236833ee6eaa24586f1a42100e39274686289e007"} Oct 31 22:49:21.338056 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.338025 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="50a20a5596d891021ceb838721b43a37aa1c3ec236a2bdd58df13aba27dea98b" exitCode=0 Oct 31 22:49:21.338150 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.338072 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"50a20a5596d891021ceb838721b43a37aa1c3ec236a2bdd58df13aba27dea98b"} Oct 31 22:49:21.353219 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:21.353163 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wzqjr" podStartSLOduration=1.896404585 podStartE2EDuration="10.353147091s" totalImagesPullingTime="8.456742506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:21.35307983 +0000 UTC m=+10.869934608" watchObservedRunningTime="2026-10-31 22:49:21.353147091 +0000 UTC m=+10.870001868" Oct 31 22:49:22.094227 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.094196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:22.094374 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.094244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:22.094374 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:22.094345 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:22.094622 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:22.094561 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:22.343812 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.343729 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="b7f51b9a6c907d1d9c069cbb4b1851a9ad52d553573c1a289703eedf1e5be41e" exitCode=0 Oct 31 22:49:22.343812 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.343799 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"b7f51b9a6c907d1d9c069cbb4b1851a9ad52d553573c1a289703eedf1e5be41e"} Oct 31 22:49:22.345877 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.345855 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wlwnn" event={"ID":"39248804-861a-4442-a717-66a2467e2c72","Type":"ContainerStarted","Data":"5cb6fe15bfdea1062c47ff62b31fb193ed05e6699ce25204b1bf245a05f4a77f"} Oct 31 22:49:22.374684 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:22.374619 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rdsnf" podStartSLOduration=3.330826974 podStartE2EDuration="11.374601259s" totalImagesPullingTime="8.043774285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:21.407580217 +0000 UTC m=+10.924434993" watchObservedRunningTime="2026-10-31 22:49:22.374601259 +0000 UTC m=+11.891456035" Oct 31 22:49:23.094590 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.094345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:23.094763 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:23.094729 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:23.350435 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.350347 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="63355b47bc1c40fc865fb4bd14c01650910aa0da64edc991f650b80d3ba72c0c" exitCode=0 Oct 31 22:49:23.350854 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.350424 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"63355b47bc1c40fc865fb4bd14c01650910aa0da64edc991f650b80d3ba72c0c"} Oct 31 22:49:23.352166 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.352144 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" event={"ID":"f91b6ba2-b532-47ef-b1de-ea7a018f2355","Type":"ContainerStarted","Data":"8875a0d30647f03187660c37e4c43a489823b30f9743c0be93788d89f5c64592"} Oct 31 22:49:23.383306 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.383259 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wlwnn" podStartSLOduration=4.009267792 podStartE2EDuration="12.38324193s" totalImagesPullingTime="8.373974138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:22.393827802 +0000 UTC m=+11.910682578" watchObservedRunningTime="2026-10-31 22:49:23.38324193 +0000 UTC m=+12.900096705" Oct 31 22:49:23.404722 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:23.404664 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" podStartSLOduration=2.093830069 podStartE2EDuration="12.404650167s" totalImagesPullingTime="10.310820098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:23.404342951 +0000 UTC m=+12.921197719" watchObservedRunningTime="2026-10-31 22:49:23.404650167 +0000 UTC m=+12.921504942" Oct 31 22:49:24.094474 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:24.094438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:24.094474 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:24.094463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:24.094730 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:24.094578 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:24.094730 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:24.094717 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:25.094355 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:25.094316 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:25.094967 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:25.094453 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:26.094219 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.094191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:26.094400 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.094200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:26.094400 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:26.094289 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:26.094400 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:26.094386 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:26.362330 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.362244 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="a047ca7df029db8d90aad97d0975de39a7085a704c26306d976013904a97ed14" exitCode=0 Oct 31 22:49:26.362330 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.362308 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"a047ca7df029db8d90aad97d0975de39a7085a704c26306d976013904a97ed14"} Oct 31 22:49:26.735203 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.735116 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:26.735358 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.735302 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:26.735692 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:26.735669 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:49:27.094339 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.094180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:27.094447 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.094419 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:27.367904 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.367877 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:49:27.368278 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368250 2582 generic.go:356] "Generic (PLEG): container finished" podID="c9fe7d02-92f2-41fd-9122-abdfdc159006" containerID="7f9b300ed9a67813c080942d4a28d0d4a0048cff0ddf9b41f6c28e0e96629014" exitCode=1 Oct 31 22:49:27.368398 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368313 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"c48bb8f667777647575e1f542a5d296c90d612e632fb857690f3001ba8cb1628"} Oct 31 22:49:27.368398 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368387 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"05d9a539352716fbd0beeb18e9b967a891a8393adfd47dedb7ab81edc06e70dc"} Oct 31 22:49:27.368500 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368428 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"383b467664ad950a037e6617f7ed515a8a52f1a3d69b0d82713e7f0cdb73e73c"} Oct 31 22:49:27.368500 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368475 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"49394a8fbc6f94f3c4a5a28a4ead1ec96439ee7a3b82bdac58bc033ec08bbd2a"} Oct 31 22:49:27.368584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368513 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerDied","Data":"7f9b300ed9a67813c080942d4a28d0d4a0048cff0ddf9b41f6c28e0e96629014"} Oct 31 22:49:27.368584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.368544 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"da005a355972663c08f082a6312d8cfb6e8d06ff8377b87fc91be5b1b57c9740"} Oct 31 22:49:27.371505 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.371480 2582 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="f4a4eeaffd81bb3bb4d0eb980b83107da7c42253b77be43f2e12bfa863a7e18c" exitCode=0 Oct 31 22:49:27.371594 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.371561 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerDied","Data":"f4a4eeaffd81bb3bb4d0eb980b83107da7c42253b77be43f2e12bfa863a7e18c"} Oct 31 22:49:27.751396 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.751208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:27.751586 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.751407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:27.751586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.751351 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:27.751586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.751510 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:27.751586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.751516 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:43.751499594 +0000 UTC m=+33.268354361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:27.751586 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.751556 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:43.751545066 +0000 UTC m=+33.268399819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:27.852420 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:27.852379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:27.852667 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.852510 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:27.852667 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.852526 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:27.852667 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.852535 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:27.852667 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:27.852577 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:49:43.852566162 +0000 UTC m=+33.369420931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:28.093844 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:28.093752 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:28.094017 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:28.093857 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:28.094017 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:28.093907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:28.094109 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:28.094013 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:28.376900 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:28.376799 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" event={"ID":"98b6f6c7-3878-491d-8a05-131c56390f37","Type":"ContainerStarted","Data":"bfbdb6c78ae6bed3a106b8153270e714a66a1764620e09c4f2de69ff9bd74b76"} Oct 31 22:49:28.407639 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:28.407585 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ctfvq" podStartSLOduration=6.935341548 podStartE2EDuration="17.407570308s" totalImagesPullingTime="10.47222876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:28.407295826 +0000 UTC m=+17.924150601" watchObservedRunningTime="2026-10-31 22:49:28.407570308 +0000 UTC m=+17.924425083" Oct 31 22:49:29.094433 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:29.094397 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:29.094619 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:29.094517 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:30.093767 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:30.093727 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:30.094341 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:30.093837 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:30.094341 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:30.093898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:30.094341 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:30.094043 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:30.382444 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:30.382355 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" event={"ID":"48ac9bdf-c637-46e4-9fac-53c43f3fd161","Type":"ContainerStarted","Data":"132fe41a23d126e80e10a63d2fd996688f4368cbe19b53e7c61cee504f7e48fa"} Oct 31 22:49:30.385288 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:30.385266 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:49:30.385610 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:30.385590 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"c05f5afbacb43d4ab42735764fd93101ac2d09488a3f0038e1b08880858df642"} Oct 31 22:49:31.002234 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.002210 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 22:49:31.095024 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.094975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:31.095587 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:31.095079 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:31.389903 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.389823 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" event={"ID":"48ac9bdf-c637-46e4-9fac-53c43f3fd161","Type":"ContainerStarted","Data":"8e2013eb4317521d90bdc5134d8e7b6e05e10d280f564d2a47e66e8b6f12513f"} Oct 31 22:49:31.977068 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.976754 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:49:31.00223125Z","UUID":"0b3ff95c-41ae-4762-8e81-d3529e2df020","Handler":null,"Name":"","Endpoint":""} Oct 31 22:49:31.978127 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.978109 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 22:49:31.978127 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:31.978131 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 22:49:32.093571 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.093488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:32.093571 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.093517 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:32.093748 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:32.093596 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:32.093748 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:32.093652 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:32.394228 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.394143 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" event={"ID":"48ac9bdf-c637-46e4-9fac-53c43f3fd161","Type":"ContainerStarted","Data":"d34d2a3f104b07a2d0e83b9e334b520df25c670b26accea3fdfcf03cab082c43"} Oct 31 22:49:32.397346 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.397324 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:49:32.397663 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.397643 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"e58437113d43fe34059592f60e1731871d307ab74149a3e10b7f54aaa31acdcf"} Oct 31 22:49:32.397889 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.397869 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:32.398011 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.397935 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:32.398083 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.398012 2582 scope.go:122] "RemoveContainer" containerID="7f9b300ed9a67813c080942d4a28d0d4a0048cff0ddf9b41f6c28e0e96629014" Oct 31 22:49:32.414119 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.414097 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:32.416932 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:32.416890 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" podStartSLOduration=1.978484523 podStartE2EDuration="21.416877971s" totalImagesPullingTime="19.438393448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-10-31 22:49:32.416719415 +0000 UTC m=+21.933574191" watchObservedRunningTime="2026-10-31 22:49:32.416877971 +0000 UTC m=+21.933732745" Oct 31 22:49:33.093645 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.093604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:33.093881 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:33.093809 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:33.403304 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.403231 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:49:33.403784 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.403587 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" event={"ID":"c9fe7d02-92f2-41fd-9122-abdfdc159006","Type":"ContainerStarted","Data":"5fe371517f3c2970341ee66a184a47b0a49b1afaff4d051450450e21ceb84989"} Oct 31 22:49:33.403911 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.403893 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:33.419138 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.419111 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:49:33.443019 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:33.442959 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" podStartSLOduration=7.831949252 podStartE2EDuration="22.442947107s" totalImagesPullingTime="14.610997855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:33.440302708 +0000 UTC m=+22.957157482" watchObservedRunningTime="2026-10-31 22:49:33.442947107 +0000 UTC m=+22.959801882" Oct 31 22:49:34.093479 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:34.093451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:34.093675 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:34.093451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:34.093675 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:34.093561 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:34.093793 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:34.093678 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:34.094034 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:34.094020 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wzqjr_25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/dns-node-resolver/0.log" Oct 31 22:49:34.666026 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:34.666000 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44szm_4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/node-ca/0.log" Oct 31 22:49:35.093743 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:35.093654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:35.093961 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:35.093762 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:36.094247 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:36.094202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:36.094247 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:36.094231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:36.094703 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:36.094320 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:36.094703 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:36.094437 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:37.094205 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:37.094171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:37.094372 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:37.094280 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:38.094217 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:38.094178 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:38.094217 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:38.094221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:38.094708 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:38.094292 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:38.094708 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:38.094406 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:39.093952 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:39.093920 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:39.094156 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:39.094042 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:40.093859 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:40.093821 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:40.093859 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:40.093850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:40.094326 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:40.093936 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:40.094326 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:40.094066 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:41.095219 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:41.095186 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:41.095724 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:41.095278 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:42.093516 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:42.093486 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:42.093679 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:42.093591 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:42.093679 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:42.093646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:42.093762 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:42.093744 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:43.093612 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:43.093579 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:43.094034 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.093682 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:43.767962 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:43.767927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:43.768227 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:43.768012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:43.768227 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.768095 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:43.768227 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.768150 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs podName:7ca1b181-263f-43a9-96fb-2a2efbd8c963 nodeName:}" failed. No retries permitted until 2026-10-31 22:50:15.768135681 +0000 UTC m=+65.284990435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs") pod "network-metrics-daemon-v5ftn" (UID: "7ca1b181-263f-43a9-96fb-2a2efbd8c963") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:43.768227 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.768095 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:43.768227 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.768192 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret podName:56499b86-c2fe-429c-88b8-cc5be0e63840 nodeName:}" failed. No retries permitted until 2026-10-31 22:50:15.768183069 +0000 UTC m=+65.285037825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret") pod "global-pull-secret-syncer-qqjd2" (UID: "56499b86-c2fe-429c-88b8-cc5be0e63840") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:43.868628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:43.868596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:43.868776 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.868725 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:43.868776 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.868738 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:43.868776 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.868746 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dzzxd for pod openshift-network-diagnostics/network-check-target-nvttn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:43.868871 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:43.868794 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd podName:e383e48a-7ee2-4c25-a5e4-d708af31e31b nodeName:}" failed. No retries permitted until 2026-10-31 22:50:15.868779901 +0000 UTC m=+65.385634668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzzxd" (UniqueName: "kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd") pod "network-check-target-nvttn" (UID: "e383e48a-7ee2-4c25-a5e4-d708af31e31b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:44.094226 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:44.094131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:44.094226 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:44.094163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:44.094706 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:44.094263 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:44.094706 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:44.094367 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:45.094062 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:45.094028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:45.094224 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:45.094161 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:46.094184 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:46.094154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:46.094652 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:46.094276 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:46.094652 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:46.094319 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:46.094652 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:46.094403 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:47.093678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:47.093638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:47.093866 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:47.093748 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:48.094372 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:48.094338 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:48.094823 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:48.094438 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:48.094823 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:48.094499 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:48.094823 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:48.094627 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:49.093984 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:49.093947 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:49.094141 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:49.094114 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:50.094004 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:50.093961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:50.094380 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:50.093961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:50.094380 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:50.094072 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:50.094380 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:50.094174 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:51.097295 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.097062 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:51.097748 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:51.097389 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:51.245294 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.245222 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nvttn"] Oct 31 22:49:51.248141 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.248115 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5ftn"] Oct 31 22:49:51.248275 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.248237 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:51.248381 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:51.248361 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:51.249178 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.249155 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqjd2"] Oct 31 22:49:51.249291 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.249255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:51.249378 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:51.249358 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:51.441338 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:51.441306 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:51.441699 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:51.441673 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:53.093818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:53.093783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:53.094381 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:53.093794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:53.094381 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:53.093877 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nvttn" podUID="e383e48a-7ee2-4c25-a5e4-d708af31e31b" Oct 31 22:49:53.094381 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:53.093964 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5ftn" podUID="7ca1b181-263f-43a9-96fb-2a2efbd8c963" Oct 31 22:49:53.094381 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:53.094041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:53.094381 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:49:53.094122 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqjd2" podUID="56499b86-c2fe-429c-88b8-cc5be0e63840" Oct 31 22:49:54.817741 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.817710 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeReady" Oct 31 22:49:54.818222 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.817834 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 22:49:54.871309 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.871281 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j2gk2"] Oct 31 22:49:54.873096 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.873079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:54.874346 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.874326 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t48z9"] Oct 31 22:49:54.875792 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.875770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:54.878771 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.878735 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 22:49:54.878868 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.878846 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 22:49:54.878933 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.878912 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 22:49:54.878980 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.878951 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 22:49:54.879107 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.879091 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9xrd\"" Oct 31 22:49:54.884345 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.884330 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 22:49:54.887518 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.887499 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2gk2"] Oct 31 22:49:54.896367 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.896346 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qbfrp"] Oct 31 22:49:54.899628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.898728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:54.902727 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.902709 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t48z9"] Oct 31 22:49:54.913592 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.913557 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qbfrp"] Oct 31 22:49:54.979587 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979458 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 22:49:54.979587 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979471 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 22:49:54.979775 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979678 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 22:49:54.979775 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979698 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7g25v\"" Oct 31 22:49:54.979775 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979712 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s2g2v\"" Oct 31 22:49:54.979775 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:54.979745 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 22:49:55.048139 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.048246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.048246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.048317 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.048317 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048280 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6jwb\" (UniqueName: \"kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.048317 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.048414 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.048451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54qb\" (UniqueName: \"kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.048484 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.048520 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpck\" (UniqueName: \"kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.048561 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.048548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.093425 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.093394 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:49:55.093514 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.093402 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:49:55.093514 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.093501 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:49:55.098246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.098221 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 22:49:55.098246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.098231 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\"" Oct 31 22:49:55.098246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.098222 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 22:49:55.098523 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.098271 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 22:49:55.100162 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.100141 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 22:49:55.100268 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.100189 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\"" Oct 31 22:49:55.100413 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.100404 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\"" Oct 31 22:49:55.149416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.149416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jwb\" (UniqueName: \"kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54qb\" (UniqueName: \"kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpck\" (UniqueName: \"kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.149633 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.149896 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.149896 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149771 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.149896 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.149896 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.149871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.150183 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.150155 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.150183 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.150175 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.150400 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.150382 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.150456 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.150385 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.153283 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.153264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.153404 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.153386 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.153442 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.153396 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.162438 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.162414 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jwb\" (UniqueName: \"kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.162631 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.162609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54qb\" (UniqueName: \"kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.162710 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.162696 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpck\" (UniqueName: \"kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.186508 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.186480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:49:55.191075 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.191055 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:49:55.210146 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.210114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:55.479538 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.479498 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t48z9"] Oct 31 22:49:55.483343 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.483316 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2gk2"] Oct 31 22:49:55.486220 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:55.486199 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb149490_9112_463a_b5ef_107f84f04d0c.slice/crio-de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b WatchSource:0}: Error finding container de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b: Status 404 returned error can't find the container with id de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b Oct 31 22:49:55.499403 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:55.499384 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qbfrp"] Oct 31 22:49:55.502251 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:55.502224 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9004f5b_8f45_4dfe_b883_d485d3b43582.slice/crio-9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388 WatchSource:0}: Error finding container 9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388: Status 404 returned error can't find the container with id 9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388 Oct 31 22:49:56.357040 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.356954 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wzqjr_25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/dns-node-resolver/0.log" Oct 31 22:49:56.454429 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.454389 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qbfrp" event={"ID":"d9004f5b-8f45-4dfe-b883-d485d3b43582","Type":"ContainerStarted","Data":"9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388"} Oct 31 22:49:56.455891 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.455862 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t48z9" event={"ID":"26fd3b3c-9c2e-43b7-9660-ec1ff811348e","Type":"ContainerStarted","Data":"43e9514ffbd7f79e2a39e3a3aa62a935084a55756861a4deb1ad0a9759631cbb"} Oct 31 22:49:56.458697 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.458672 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2gk2" event={"ID":"eb149490-9112-463a-b5ef-107f84f04d0c","Type":"ContainerStarted","Data":"455d04d8fb6a6d42f8be1d1ccee0d81f4837391167493c72643baaa187e48aef"} Oct 31 22:49:56.458800 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.458723 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2gk2" event={"ID":"eb149490-9112-463a-b5ef-107f84f04d0c","Type":"ContainerStarted","Data":"814a37204d107a948f7e1a5f1e5fb736d806f69b043356e526bd9c1866d3d57d"} Oct 31 22:49:56.458800 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:56.458750 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2gk2" event={"ID":"eb149490-9112-463a-b5ef-107f84f04d0c","Type":"ContainerStarted","Data":"de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b"} Oct 31 22:49:57.156584 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.156557 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44szm_4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/node-ca/0.log" Oct 31 22:49:57.463326 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.463276 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qbfrp" event={"ID":"d9004f5b-8f45-4dfe-b883-d485d3b43582","Type":"ContainerStarted","Data":"8e89bc16588dedb2f3d6acd2687ec7281a71dc29e8f0ddf246912d318c89ae1f"} Oct 31 22:49:57.769031 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.768984 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:49:57.771508 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.771482 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.777310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.777288 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 22:49:57.777431 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.777389 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 22:49:57.777431 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.777292 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 22:49:57.779013 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.778770 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 22:49:57.779013 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.778847 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 22:49:57.779013 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.778907 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cbxkt\"" Oct 31 22:49:57.779439 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.779319 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 22:49:57.787078 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.787055 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 22:49:57.787193 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.787102 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 22:49:57.787835 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.787816 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:49:57.872847 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.872826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.872955 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.872854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.872955 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.872875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.873057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.872998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.873057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.873023 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.873155 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.873062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.873155 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.873140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.873229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.873167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974217 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974375 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974375 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974375 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974375 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974591 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974591 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974685 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.974828 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.974802 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.975056 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.975036 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.975126 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.975036 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.975247 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.975227 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.975463 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.975444 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.977436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.977414 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.977567 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.977509 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:57.987870 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:57.987847 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:58.083080 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.082973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:49:58.363928 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.363902 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:49:58.365746 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:58.365720 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc12ea2_4cb4_4f1f_9b6b_157c5b0c403e.slice/crio-6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397 WatchSource:0}: Error finding container 6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397: Status 404 returned error can't find the container with id 6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397 Oct 31 22:49:58.467951 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.467917 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qbfrp" event={"ID":"d9004f5b-8f45-4dfe-b883-d485d3b43582","Type":"ContainerStarted","Data":"9dcf5a8195e945129d991ad5b08cb504eb27b9ed0067cecff2871a98ef9ee70f"} Oct 31 22:49:58.468390 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.468009 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qbfrp" Oct 31 22:49:58.469955 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.469929 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2gk2" event={"ID":"eb149490-9112-463a-b5ef-107f84f04d0c","Type":"ContainerStarted","Data":"880f7da73da07ced28dbed5720be226a4a14b29cf61b9ff693ba3ccf43e92f20"} Oct 31 22:49:58.471027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.471008 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6f844c96-s75ms" event={"ID":"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e","Type":"ContainerStarted","Data":"6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397"} Oct 31 22:49:58.472353 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.472335 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t48z9" event={"ID":"26fd3b3c-9c2e-43b7-9660-ec1ff811348e","Type":"ContainerStarted","Data":"8330178f3af79905aa26501e22842ce940a3a2873df3f202edab6398d52cb95c"} Oct 31 22:49:58.490529 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.490489 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qbfrp" podStartSLOduration=2.695134518 podStartE2EDuration="4.490477929s" totalImagesPullingTime="1.795343411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:58.489931786 +0000 UTC m=+48.006786563" watchObservedRunningTime="2026-10-31 22:49:58.490477929 +0000 UTC m=+48.007332704" Oct 31 22:49:58.516216 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.516176 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t48z9" podStartSLOduration=1.830889729 podStartE2EDuration="4.516164507s" totalImagesPullingTime="2.685274778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:58.515491188 +0000 UTC m=+48.032345963" watchObservedRunningTime="2026-10-31 22:49:58.516164507 +0000 UTC m=+48.033019281" Oct 31 22:49:58.546190 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.546152 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j2gk2" podStartSLOduration=2.365900524 podStartE2EDuration="4.546143099s" totalImagesPullingTime="2.180242575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:58.545967681 +0000 UTC m=+48.062822456" watchObservedRunningTime="2026-10-31 22:49:58.546143099 +0000 UTC m=+48.062997873" Oct 31 22:49:58.811629 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.811596 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq"] Oct 31 22:49:58.813686 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.813666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.824830 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.824808 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 22:49:58.825049 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.824978 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 22:49:58.825246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.825221 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 22:49:58.825246 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.825227 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 22:49:58.825393 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.825260 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6c492\"" Oct 31 22:49:58.825393 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.825243 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 22:49:58.826646 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.826626 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7wpwb"] Oct 31 22:49:58.831747 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.831723 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq"] Oct 31 22:49:58.832338 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.832309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.840641 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.840618 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc"] Oct 31 22:49:58.842333 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.842314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.857930 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.857904 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc"] Oct 31 22:49:58.881588 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.881564 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 22:49:58.881665 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.881609 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 22:49:58.882133 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.882118 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 22:49:58.882229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.882219 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 22:49:58.882268 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.882251 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kvll7\"" Oct 31 22:49:58.883036 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883021 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p9kg6\"" Oct 31 22:49:58.883399 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883444 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883444 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883544 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883499 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcqm\" (UniqueName: \"kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.883544 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883625 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.883655 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.883655 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvh7r\" (UniqueName: \"kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.883732 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.883732 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbn7\" (UniqueName: \"kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883732 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.883832 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883865 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883865 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883926 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.883926 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.884039 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.884039 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.883951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.884132 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.884042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.924863 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.924827 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 22:49:58.925058 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.925041 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 22:49:58.984499 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.984632 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.984632 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.984632 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.984632 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.984846 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.984846 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984646 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.984846 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.984731 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985044 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985101 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985139 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985099 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985168 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985204 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985166 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985204 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985277 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcqm\" (UniqueName: \"kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985330 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985361 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.985392 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.985392 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvh7r\" (UniqueName: \"kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.985481 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985481 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985428 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.985481 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbn7\" (UniqueName: \"kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985616 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.985668 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985617 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.985715 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.985681 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.986279 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.986254 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.986397 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.986352 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.986920 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.986895 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.987948 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.987904 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.988471 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.988405 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.988709 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.988683 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.988972 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.988946 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.989743 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.989721 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.990136 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.990087 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:58.999186 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.999159 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbn7\" (UniqueName: \"kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:58.999828 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.999784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcqm\" (UniqueName: \"kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:58.999828 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:58.999819 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvh7r\" (UniqueName: \"kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:59.125600 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.125528 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:49:59.145536 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.145506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:49:59.154040 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.154014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:49:59.156965 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:59.156862 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d187ad0_d0cd_4442_a0b5_ab7e324cacda.slice/crio-5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf WatchSource:0}: Error finding container 5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf: Status 404 returned error can't find the container with id 5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf Oct 31 22:49:59.409901 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.409832 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq"] Oct 31 22:49:59.413341 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:59.413310 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c47d3f0_fedf_456a_b4db_dc5115e9c5d6.slice/crio-c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a WatchSource:0}: Error finding container c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a: Status 404 returned error can't find the container with id c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a Oct 31 22:49:59.440978 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.440945 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc"] Oct 31 22:49:59.447033 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:49:59.447003 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2252e42_a7dd_4822_83ca_65aff7bb802a.slice/crio-4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871 WatchSource:0}: Error finding container 4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871: Status 404 returned error can't find the container with id 4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871 Oct 31 22:49:59.476273 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.476242 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" event={"ID":"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6","Type":"ContainerStarted","Data":"c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a"} Oct 31 22:49:59.477579 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.477556 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" event={"ID":"b2252e42-a7dd-4822-83ca-65aff7bb802a","Type":"ContainerStarted","Data":"4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871"} Oct 31 22:49:59.479126 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.479099 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wpwb" event={"ID":"0d187ad0-d0cd-4442-a0b5-ab7e324cacda","Type":"ContainerStarted","Data":"5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf"} Oct 31 22:49:59.921783 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.921754 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:49:59.924119 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.924090 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.930449 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930427 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 22:49:59.930571 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930535 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 22:49:59.930808 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930789 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 22:49:59.930902 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930888 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 22:49:59.930951 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930927 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 22:49:59.931033 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.930980 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 22:49:59.931110 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.931082 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 22:49:59.931240 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.931226 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 22:49:59.931318 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.931306 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 22:49:59.931385 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.931373 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h45ml\"" Oct 31 22:49:59.946809 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.944191 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:49:59.994865 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.994813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995059 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.994888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995059 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.994923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995059 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.994947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995059 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.994978 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995059 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995243 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995243 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995243 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995243 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995430 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995430 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:49:59.995430 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:49:59.995299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096289 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096451 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.096695 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.096692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.097620 ip-10-0-131-157 kubenswrapper[2582]: E1031 22:50:00.097599 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle podName:dc61c3ce-5106-4bbd-b69d-e3aa762aba20 nodeName:}" failed. No retries permitted until 2026-10-31 22:50:00.597580392 +0000 UTC m=+50.114435164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20") : configmap references non-existent config key: ca-bundle.crt Oct 31 22:50:00.098045 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.098019 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.098541 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.098476 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.100396 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.100085 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.101676 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.101459 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.103613 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.103589 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.104279 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.104255 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.104362 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.104320 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.104503 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.104483 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.104503 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.104496 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.106314 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.106290 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.107096 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.107062 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.111628 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.111371 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.485911 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.485869 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wpwb" event={"ID":"0d187ad0-d0cd-4442-a0b5-ab7e324cacda","Type":"ContainerStarted","Data":"a52fd81782f8af33190d392014919f893337bf29d8f413465f098a0eba6dd37e"} Oct 31 22:50:00.489229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.489117 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" event={"ID":"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6","Type":"ContainerStarted","Data":"3c8ee8bc2543840ff144ce907bbbaa5979a9d6a6c0f32c53c961c675632ddb34"} Oct 31 22:50:00.489229 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.489178 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" event={"ID":"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6","Type":"ContainerStarted","Data":"356b136f58e6c0fb94bdc83d2ab623f6c8001f653bbe9f5afdbbd009145a62c3"} Oct 31 22:50:00.600258 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.600179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.601607 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.601582 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.847123 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.847088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:00.899203 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.899168 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5896fc6d85-lpp69"] Oct 31 22:50:00.903596 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.902318 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.910749 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.910853 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.910932 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.912588 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qlpwp\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.912623 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.912701 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b307j9jpi8iol\"" Oct 31 22:50:00.913342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.910982 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 22:50:00.924213 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:00.924142 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5896fc6d85-lpp69"] Oct 31 22:50:01.003801 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.003769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.003942 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.003891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004037 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.003952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004037 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.003982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.004039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.004074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004357 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.004203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004357 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.004244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8ps\" (UniqueName: \"kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.004357 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.004285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105290 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8ps\" (UniqueName: \"kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.105818 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.106037 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.105957 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.108262 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.108228 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.108538 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.108511 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.108646 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.108579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.108981 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.108958 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.109690 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.109670 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.109774 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.109758 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.110207 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.110189 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.120359 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.120336 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8ps\" (UniqueName: \"kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.144214 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.144194 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:01.148450 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:01.148351 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc61c3ce_5106_4bbd_b69d_e3aa762aba20.slice/crio-b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468 WatchSource:0}: Error finding container b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468: Status 404 returned error can't find the container with id b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468 Oct 31 22:50:01.257406 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.257381 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:01.496139 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.496100 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" event={"ID":"b2252e42-a7dd-4822-83ca-65aff7bb802a","Type":"ContainerStarted","Data":"14d7cffd2ce317c03983b05fb4afcf0b1db16fc532b2548b10643f62e2f151e1"} Oct 31 22:50:01.496565 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.496171 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" event={"ID":"b2252e42-a7dd-4822-83ca-65aff7bb802a","Type":"ContainerStarted","Data":"abdf59559dcda16e240541d8c449bb1e7a3d1cfd83e55e0aa4d36ed4e220b8ca"} Oct 31 22:50:01.496565 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.496200 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" event={"ID":"b2252e42-a7dd-4822-83ca-65aff7bb802a","Type":"ContainerStarted","Data":"ea2cf5c7c655f24f9b619f91606b604722970bfffe66891b126bd85fe6d8da81"} Oct 31 22:50:01.498185 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.498158 2582 generic.go:356] "Generic (PLEG): container finished" podID="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" containerID="a52fd81782f8af33190d392014919f893337bf29d8f413465f098a0eba6dd37e" exitCode=0 Oct 31 22:50:01.498294 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.498250 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wpwb" event={"ID":"0d187ad0-d0cd-4442-a0b5-ab7e324cacda","Type":"ContainerDied","Data":"a52fd81782f8af33190d392014919f893337bf29d8f413465f098a0eba6dd37e"} Oct 31 22:50:01.500835 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.500805 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" event={"ID":"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6","Type":"ContainerStarted","Data":"b5c3d38ace41ff4e7fa5812485b328fe52ff7ebd1a6d5c1a347740f9ce8117fb"} Oct 31 22:50:01.502080 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.502050 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468"} Oct 31 22:50:01.521088 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.520948 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" podStartSLOduration=1.824687503 podStartE2EDuration="3.520931836s" totalImagesPullingTime="1.696244333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:01.520231884 +0000 UTC m=+51.037086656" watchObservedRunningTime="2026-10-31 22:50:01.520931836 +0000 UTC m=+51.037786613" Oct 31 22:50:01.631844 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.631814 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5896fc6d85-lpp69"] Oct 31 22:50:01.633105 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:01.633083 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dc71e0_8a72_4a66_a4ab_cb5c3e462d4f.slice/crio-62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae WatchSource:0}: Error finding container 62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae: Status 404 returned error can't find the container with id 62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae Oct 31 22:50:01.644342 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:01.644296 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" podStartSLOduration=2.473128148 podStartE2EDuration="3.644283127s" totalImagesPullingTime="1.171154979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:01.642501573 +0000 UTC m=+51.159356342" watchObservedRunningTime="2026-10-31 22:50:01.644283127 +0000 UTC m=+51.161137894" Oct 31 22:50:02.507235 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.507199 2582 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c" exitCode=0 Oct 31 22:50:02.507723 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.507293 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c"} Oct 31 22:50:02.510260 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.510229 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wpwb" event={"ID":"0d187ad0-d0cd-4442-a0b5-ab7e324cacda","Type":"ContainerStarted","Data":"cbc45a5b33ac94f0857fe00cd29f66e90a4aa3c1c82c7609206d2cfeb31adc58"} Oct 31 22:50:02.510353 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.510278 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wpwb" event={"ID":"0d187ad0-d0cd-4442-a0b5-ab7e324cacda","Type":"ContainerStarted","Data":"6a24530bf2568f6a9b8e310a3ad6357b07ca13bbb50a111cb9924695ac2eaa26"} Oct 31 22:50:02.512156 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.512125 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6f844c96-s75ms" event={"ID":"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e","Type":"ContainerStarted","Data":"a946ee60240b8a747f2611cb0f734552efa513073ed6f100aa9224df8f3c8205"} Oct 31 22:50:02.513643 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.513613 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae"} Oct 31 22:50:02.570056 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.569998 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7wpwb" podStartSLOduration=3.330625874 podStartE2EDuration="4.56996819s" totalImagesPullingTime="1.16262606s" totalInitContainerRuntime="76.716256ms" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:02.56870201 +0000 UTC m=+52.085556786" watchObservedRunningTime="2026-10-31 22:50:02.56996819 +0000 UTC m=+52.086822968" Oct 31 22:50:02.594720 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:02.594667 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c6f844c96-s75ms" podStartSLOduration=2.393270997 podStartE2EDuration="5.594651139s" totalImagesPullingTime="3.201380142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:02.593724401 +0000 UTC m=+52.110579175" watchObservedRunningTime="2026-10-31 22:50:02.594651139 +0000 UTC m=+52.111505913" Oct 31 22:50:03.214417 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.214381 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d849c4dfc-tbtqx"] Oct 31 22:50:03.217804 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.217778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.222608 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.222583 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 22:50:03.222714 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.222700 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 22:50:03.229024 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.228945 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d849c4dfc-tbtqx"] Oct 31 22:50:03.257191 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.257146 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 22:50:03.257371 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.257206 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7phlcd127qi9k\"" Oct 31 22:50:03.257371 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.257230 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 22:50:03.257371 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.257334 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2whgz\"" Oct 31 22:50:03.327727 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.327898 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.327898 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.327898 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.327898 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4sl8\" (UniqueName: \"kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.328153 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.328153 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.327980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429292 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429455 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429455 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429455 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4sl8\" (UniqueName: \"kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429455 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429669 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.429669 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.429478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.430724 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.430400 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.430724 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.430527 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.431135 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.431108 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.432880 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.432858 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.434278 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.434236 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.434278 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.434265 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.443592 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.443571 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4sl8\" (UniqueName: \"kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.533403 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.533369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:03.617496 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.616557 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p"] Oct 31 22:50:03.621071 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.620676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.626588 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.626544 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n6lmz\"" Oct 31 22:50:03.626710 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.626586 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 22:50:03.635437 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.634751 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p"] Oct 31 22:50:03.732173 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.732140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.732306 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.732189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxxz\" (UniqueName: \"kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.786038 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.785953 2582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:50:03.818109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.818083 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:50:03.822664 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.822644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.828485 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.828458 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d849c4dfc-tbtqx"] Oct 31 22:50:03.831551 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:03.831522 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9ed1c1_75ac_42cc_8613_67993c17b9a5.slice/crio-946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6 WatchSource:0}: Error finding container 946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6: Status 404 returned error can't find the container with id 946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6 Oct 31 22:50:03.834397 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.834372 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:50:03.835170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.834981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.835170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.835062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxxz\" (UniqueName: \"kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.837926 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.837906 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.849150 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.849122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxxz\" (UniqueName: \"kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:03.935629 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935733 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935792 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935850 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935889 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935856 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935923 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.935969 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.935925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.936155 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.936137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:03.944420 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:03.944390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:04.037464 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037464 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037432 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.037670 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.037597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.038306 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.038256 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.038406 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.038385 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.038463 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.038428 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.038761 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.038728 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.039310 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.039269 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.040509 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.040488 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.040582 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.040527 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.051423 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.051399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.154977 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.154937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:04.233062 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.227497 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p"] Oct 31 22:50:04.239396 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:04.239363 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c3dce2_82b9_4868_8ec9_2d480ff15a1c.slice/crio-1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb WatchSource:0}: Error finding container 1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb: Status 404 returned error can't find the container with id 1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb Oct 31 22:50:04.454152 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.454116 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:50:04.460442 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:04.460416 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda737f063_f6c6_4120_bd76_2b666dd9b154.slice/crio-3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a WatchSource:0}: Error finding container 3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a: Status 404 returned error can't find the container with id 3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a Oct 31 22:50:04.527339 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.527299 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"0a695baefb47fc8be5db49a299462179221b8de03209770ab3c74048421a21ab"} Oct 31 22:50:04.527470 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.527387 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"d32e8793f7d15b3ff7746fe149ffa442d688d26663a1a26ec2157402ed0f0346"} Oct 31 22:50:04.527470 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.527428 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"1f2354b9bc33a54ba23980e6f6db6421ce1695ffda5ff3d023753587f288a5bc"} Oct 31 22:50:04.532262 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.530024 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6f5dbdb9-6wwq7" event={"ID":"a737f063-f6c6-4120-bd76-2b666dd9b154","Type":"ContainerStarted","Data":"3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a"} Oct 31 22:50:04.533309 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.533272 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" event={"ID":"cc9ed1c1-75ac-42cc-8613-67993c17b9a5","Type":"ContainerStarted","Data":"946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6"} Oct 31 22:50:04.539701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.539447 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa"} Oct 31 22:50:04.539701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.539515 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d"} Oct 31 22:50:04.539701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.539540 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1"} Oct 31 22:50:04.539701 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.539562 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71"} Oct 31 22:50:04.544103 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.542098 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" event={"ID":"27c3dce2-82b9-4868-8ec9-2d480ff15a1c","Type":"ContainerStarted","Data":"1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb"} Oct 31 22:50:04.558053 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:04.556588 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d6f5dbdb9-6wwq7" podStartSLOduration=1.556568928 podStartE2EDuration="1.556568928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:04.555940316 +0000 UTC m=+54.072795094" watchObservedRunningTime="2026-10-31 22:50:04.556568928 +0000 UTC m=+54.073423704" Oct 31 22:50:05.291303 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.289124 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:05.293625 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.293596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.307536 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.307512 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 22:50:05.307725 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.307707 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 22:50:05.307807 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.307795 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 22:50:05.307871 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.307857 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aej5qsglgbkq\"" Oct 31 22:50:05.307940 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.307929 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 22:50:05.311559 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.311536 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 22:50:05.317527 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.317502 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:05.318080 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.317924 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 22:50:05.318080 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.318055 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 22:50:05.318677 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.318651 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 22:50:05.318854 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.318831 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 22:50:05.323898 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.323871 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349110 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349280 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349306 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350027 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350721 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350721 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350721 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.350721 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.349547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.400770 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.400738 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 22:50:05.400967 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.400953 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9d5qn\"" Oct 31 22:50:05.401068 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.400956 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 22:50:05.401068 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.401025 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 22:50:05.401153 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.401075 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 22:50:05.421083 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.421048 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:05.451078 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451234 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451234 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451234 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451234 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451416 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451605 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451605 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451605 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451605 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451605 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451777 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.451861 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.451833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.452187 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.452169 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.453190 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.452852 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.454335 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.453974 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.454794 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.454770 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.455384 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.455045 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.455384 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.455171 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.455758 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.455663 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.455919 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.455894 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.456082 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.455927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.456534 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.456479 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.456945 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.456894 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477530 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477499 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477493 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477560 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477646 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477678 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477661 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.477885 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.477798 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.478135 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.478092 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.478228 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.478182 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.483449 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.483421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.557063 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.557024 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93"} Oct 31 22:50:05.557442 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.557142 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerStarted","Data":"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a"} Oct 31 22:50:05.561233 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.561114 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"53affff37dc7f4770385b63c1e75419e7e9c344df3d028f4c20afcfeab8e61db"} Oct 31 22:50:05.561233 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.561184 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"b6f32d07a14fd96e3335081d8b540ec7b8a39a6bea8a74ce03748a476df8bc34"} Oct 31 22:50:05.561233 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.561218 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" event={"ID":"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f","Type":"ContainerStarted","Data":"af0f08d5bd786f7d9039d9970b3e4b0ee5d552bd952ee83d1d928dad33129bcf"} Oct 31 22:50:05.561436 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.561356 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:05.563255 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.563228 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6f5dbdb9-6wwq7" event={"ID":"a737f063-f6c6-4120-bd76-2b666dd9b154","Type":"ContainerStarted","Data":"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9"} Oct 31 22:50:05.599048 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.598983 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4131235 podStartE2EDuration="6.598969415s" totalImagesPullingTime="3.173645646s" totalInitContainerRuntime="12.200269ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:05.598014322 +0000 UTC m=+55.114869091" watchObservedRunningTime="2026-10-31 22:50:05.598969415 +0000 UTC m=+55.115824189" Oct 31 22:50:05.627864 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.627827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:05.639088 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.639030 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" podStartSLOduration=2.550096072 podStartE2EDuration="5.638981401s" totalImagesPullingTime="3.088885329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:05.6361539 +0000 UTC m=+55.153008676" watchObservedRunningTime="2026-10-31 22:50:05.638981401 +0000 UTC m=+55.155836176" Oct 31 22:50:05.952360 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:05.952328 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:05.955731 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:05.955701 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ec9d28_478d_45f2_82f5_8197104db046.slice/crio-1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6 WatchSource:0}: Error finding container 1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6: Status 404 returned error can't find the container with id 1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6 Oct 31 22:50:06.568810 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.568779 2582 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f" exitCode=0 Oct 31 22:50:06.569259 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.568872 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f"} Oct 31 22:50:06.569259 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.568963 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6"} Oct 31 22:50:06.570615 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.570585 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" event={"ID":"cc9ed1c1-75ac-42cc-8613-67993c17b9a5","Type":"ContainerStarted","Data":"3e5c90f27a401b2c89878fdd75d21be0af50a3cc194ab3fd2b3c7c564554e9bc"} Oct 31 22:50:06.572244 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.572220 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" event={"ID":"27c3dce2-82b9-4868-8ec9-2d480ff15a1c","Type":"ContainerStarted","Data":"55c45abf051af690e1ebec2511c3c7e34e2df82adcd88364d263c9c9ebb1c8a0"} Oct 31 22:50:06.572902 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.572880 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:06.577526 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.577507 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:06.637429 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.637377 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" podStartSLOduration=1.987777158 podStartE2EDuration="3.637362246s" totalImagesPullingTime="1.649585088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:06.637156481 +0000 UTC m=+56.154011259" watchObservedRunningTime="2026-10-31 22:50:06.637362246 +0000 UTC m=+56.154217022" Oct 31 22:50:06.667244 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:06.667171 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" podStartSLOduration=1.957520191 podStartE2EDuration="3.667149606s" totalImagesPullingTime="1.709629415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:06.664869246 +0000 UTC m=+56.181724022" watchObservedRunningTime="2026-10-31 22:50:06.667149606 +0000 UTC m=+56.184004383" Oct 31 22:50:08.083713 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:08.083681 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:08.481798 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:08.481709 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:10.590180 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590147 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725"} Oct 31 22:50:10.590522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590208 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb"} Oct 31 22:50:10.590522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590234 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4"} Oct 31 22:50:10.590522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590257 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d"} Oct 31 22:50:10.590522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590283 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9"} Oct 31 22:50:10.590522 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.590305 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerStarted","Data":"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1"} Oct 31 22:50:10.628590 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.628569 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:10.637021 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:10.636960 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6397600690000003 podStartE2EDuration="5.636947141s" totalImagesPullingTime="2.982029218s" totalInitContainerRuntime="15.157854ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:10.635660332 +0000 UTC m=+60.152515109" watchObservedRunningTime="2026-10-31 22:50:10.636947141 +0000 UTC m=+60.153801916" Oct 31 22:50:11.578916 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:11.578880 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:12.095328 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:12.095299 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:12.114352 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:12.114321 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:12.614104 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:12.614075 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:14.156077 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:14.156042 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:14.156464 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:14.156103 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:14.160784 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:14.160762 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:14.610747 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:14.610717 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:15.852374 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.852338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:15.853057 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.853029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:15.858935 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.858913 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 22:50:15.858935 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.858924 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 22:50:15.866293 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.866262 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:15.866537 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.866516 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:15.954244 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.954201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:15.958337 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.958316 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 22:50:15.969137 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.969115 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 22:50:15.978170 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:15.978145 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:16.109299 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.109228 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\"" Oct 31 22:50:16.115117 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.115092 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\"" Oct 31 22:50:16.116109 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.116092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:16.118583 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.118563 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\"" Oct 31 22:50:16.121833 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.121815 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:16.125824 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.125796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:16.404908 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.404881 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqjd2"] Oct 31 22:50:16.408584 ip-10-0-131-157 kubenswrapper[2582]: W1031 22:50:16.408552 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56499b86_c2fe_429c_88b8_cc5be0e63840.slice/crio-1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476 WatchSource:0}: Error finding container 1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476: Status 404 returned error can't find the container with id 1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476 Oct 31 22:50:16.435183 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.433094 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5ftn"] Oct 31 22:50:16.435777 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.435757 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nvttn"] Oct 31 22:50:16.615856 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.615819 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nvttn" event={"ID":"e383e48a-7ee2-4c25-a5e4-d708af31e31b","Type":"ContainerStarted","Data":"3bc02dd39b5e5ba8babd046b0dc520e6a6d5548ce9e691a23a1e6b71cd4e4561"} Oct 31 22:50:16.617158 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.617127 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5ftn" event={"ID":"7ca1b181-263f-43a9-96fb-2a2efbd8c963","Type":"ContainerStarted","Data":"4eb8fac7d20461bbe7ab8068c6039524c9b69568b5aeec556efdbf38e0608232"} Oct 31 22:50:16.618406 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:16.618379 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qqjd2" event={"ID":"56499b86-c2fe-429c-88b8-cc5be0e63840","Type":"ContainerStarted","Data":"1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476"} Oct 31 22:50:18.643819 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:18.643781 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5ftn" event={"ID":"7ca1b181-263f-43a9-96fb-2a2efbd8c963","Type":"ContainerStarted","Data":"d179ebed51a480041fb6a9d9f09601f42531f14af75111fe28848e50b7f60df6"} Oct 31 22:50:18.644325 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:18.643847 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5ftn" event={"ID":"7ca1b181-263f-43a9-96fb-2a2efbd8c963","Type":"ContainerStarted","Data":"947adfd1adee46d8c689d728bb15cfe46c7e7601c7367f3df01e6c372f0bc7c8"} Oct 31 22:50:18.671393 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:18.670203 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v5ftn" podStartSLOduration=65.886197246 podStartE2EDuration="1m7.670185863s" totalImagesPullingTime="1.783988617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:18.669160012 +0000 UTC m=+68.186014801" watchObservedRunningTime="2026-10-31 22:50:18.670185863 +0000 UTC m=+68.187040638" Oct 31 22:50:20.653949 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:20.653849 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nvttn" event={"ID":"e383e48a-7ee2-4c25-a5e4-d708af31e31b","Type":"ContainerStarted","Data":"4be7c0e53cf744ccd368307409f8baf1ec0a44c8ac0fedec465aa222a44ff850"} Oct 31 22:50:20.654365 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:20.653959 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:20.674408 ip-10-0-131-157 kubenswrapper[2582]: I1031 22:50:20.674352 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nvttn" podStartSLOduration=65.74765228 podStartE2EDuration="1m9.674333555s" totalImagesPullingTime="3.926681275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:20.672886374 +0000 UTC m=+70.189741149" watchObservedRunningTime="2026-10-31 22:50:20.674333555 +0000 UTC m=+70.191188338" Oct 31 22:50:21.959926 ip-10-0-131-157 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 22:50:21.969453 ip-10-0-131-157 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 22:50:21.969684 ip-10-0-131-157 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 22:50:21.969890 ip-10-0-131-157 systemd[1]: kubelet.service: Consumed 4.718s CPU time. Oct 31 22:50:21.983589 ip-10-0-131-157 systemd[1]: Starting Kubernetes Kubelet... Oct 31 22:50:22.045927 ip-10-0-131-157 kubenswrapper[7074]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:50:22.045927 ip-10-0-131-157 kubenswrapper[7074]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 22:50:22.045927 ip-10-0-131-157 kubenswrapper[7074]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:50:22.045927 ip-10-0-131-157 kubenswrapper[7074]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:50:22.048392 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048307 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.048392 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048391 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.048392 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048394 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048397 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048402 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048405 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048408 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048414 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048416 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048419 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048422 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048424 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048427 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048430 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048432 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048440 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048443 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048445 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048455 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048457 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048460 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048463 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.048487 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048465 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048468 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048471 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048473 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048476 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048478 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048481 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048484 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048486 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048489 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048492 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048495 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048498 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048501 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048503 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048506 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048509 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048512 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048514 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.049014 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048517 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048519 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048522 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048524 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048527 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048531 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048533 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048536 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048540 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048543 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048546 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048549 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048552 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048554 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048557 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048560 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048562 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048565 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048568 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048570 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.049490 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048573 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048576 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048579 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048582 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048584 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048588 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048591 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048594 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048596 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048599 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048602 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048604 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048607 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048609 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048613 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048622 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048625 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048628 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048630 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048633 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.049980 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048635 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048638 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048641 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048643 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048646 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048648 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048651 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048653 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048656 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048658 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048661 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048663 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048666 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048669 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048672 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048674 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048677 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048680 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048683 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.050493 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048691 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048694 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048698 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048713 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048717 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.048720 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049504 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049512 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049514 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049519 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049521 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049524 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049527 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049529 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049532 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049534 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049537 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049540 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049543 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049546 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.050962 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049549 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049551 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049554 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049556 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049559 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049561 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049564 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049566 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049569 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049571 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049574 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049576 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049579 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049581 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049584 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049586 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049589 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049592 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049595 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.051476 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049597 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049600 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049603 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049606 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049609 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049611 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049614 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049616 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049619 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049622 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049625 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049627 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049630 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049632 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049636 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049638 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049641 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049643 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049646 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.051957 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049648 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049651 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049653 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049656 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049659 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049661 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049664 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049667 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049669 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049672 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049675 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049679 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049683 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049687 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049692 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049695 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049698 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049702 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049705 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.052470 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049708 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049711 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049714 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049716 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049719 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049721 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049724 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049727 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049729 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049732 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049734 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049737 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049739 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049742 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049744 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049747 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049750 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049752 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049755 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049757 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.053002 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049760 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049762 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049766 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049768 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049771 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049773 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049776 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049779 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049781 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049785 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049787 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049790 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049793 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049795 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.049799 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050332 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050340 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050343 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050346 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.053523 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050348 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050351 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050353 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050356 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050359 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050361 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050364 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050367 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050370 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050373 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050375 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050378 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050381 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050384 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050387 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050390 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050392 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050395 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050397 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050400 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.053979 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050403 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050406 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050408 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050411 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050414 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050417 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050419 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050422 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050424 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050427 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050429 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050432 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050434 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050437 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050440 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050442 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050445 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050447 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050449 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050452 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.054518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050454 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050457 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050460 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050462 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050464 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050468 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050471 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050473 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050476 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050478 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050481 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050484 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050486 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050489 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050491 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050494 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050496 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050500 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050502 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.055065 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050506 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050510 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050513 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050515 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050518 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050520 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050523 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050525 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050528 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050531 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050533 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050536 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050539 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050541 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050548 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050552 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050554 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050557 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050560 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050563 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.055547 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050565 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050568 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050571 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050574 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050576 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050579 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050581 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050584 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050588 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050591 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050594 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050599 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050601 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050604 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050607 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050610 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050612 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050615 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050618 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.056054 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050620 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050623 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050625 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.050627 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050713 7074 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050721 7074 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050726 7074 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050731 7074 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050738 7074 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050741 7074 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050746 7074 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050750 7074 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050754 7074 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050757 7074 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050761 7074 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050764 7074 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050767 7074 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050771 7074 flags.go:64] FLAG: --cgroup-root="" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050774 7074 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050777 7074 flags.go:64] FLAG: --client-ca-file="" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050780 7074 flags.go:64] FLAG: --cloud-provider="external" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050783 7074 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050788 7074 flags.go:64] FLAG: --cluster-domain="" Oct 31 22:50:22.056524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050791 7074 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050794 7074 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050798 7074 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050801 7074 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050806 7074 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050809 7074 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050812 7074 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050816 7074 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050819 7074 flags.go:64] FLAG: --contention-profiling="false" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050822 7074 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050825 7074 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050828 7074 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050831 7074 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050836 7074 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050839 7074 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050842 7074 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050845 7074 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050849 7074 flags.go:64] FLAG: --enable-server="true" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050853 7074 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050856 7074 flags.go:64] FLAG: --event-burst="100" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050860 7074 flags.go:64] FLAG: --event-qps="50" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050864 7074 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050867 7074 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050871 7074 flags.go:64] FLAG: --eviction-hard="" Oct 31 22:50:22.057101 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050875 7074 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050878 7074 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050881 7074 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050884 7074 flags.go:64] FLAG: --eviction-soft="" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050887 7074 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050890 7074 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050893 7074 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050896 7074 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050899 7074 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050902 7074 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050905 7074 flags.go:64] FLAG: --feature-gates="" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050910 7074 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050913 7074 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050916 7074 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050919 7074 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050922 7074 flags.go:64] FLAG: --healthz-port="10248" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050925 7074 flags.go:64] FLAG: --help="false" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050928 7074 flags.go:64] FLAG: --hostname-override="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050932 7074 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050935 7074 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050938 7074 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050941 7074 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050945 7074 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 22:50:22.057719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050948 7074 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050951 7074 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050955 7074 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050958 7074 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050961 7074 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050964 7074 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050968 7074 flags.go:64] FLAG: --kube-reserved="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050972 7074 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050975 7074 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050978 7074 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050981 7074 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.050984 7074 flags.go:64] FLAG: --lock-file="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051001 7074 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051006 7074 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051010 7074 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051016 7074 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051018 7074 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051022 7074 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051025 7074 flags.go:64] FLAG: --logging-format="text" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051028 7074 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051031 7074 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051034 7074 flags.go:64] FLAG: --manifest-url="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051038 7074 flags.go:64] FLAG: --manifest-url-header="" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051043 7074 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051046 7074 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 22:50:22.058291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051049 7074 flags.go:64] FLAG: --max-pods="110" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051053 7074 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051056 7074 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051059 7074 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051062 7074 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051065 7074 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051068 7074 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051070 7074 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051077 7074 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051082 7074 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051085 7074 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051088 7074 flags.go:64] FLAG: --pod-cidr="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051090 7074 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051095 7074 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051099 7074 flags.go:64] FLAG: --pods-per-core="0" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051102 7074 flags.go:64] FLAG: --port="10250" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051105 7074 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051108 7074 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ff613f49a18024d7" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051111 7074 flags.go:64] FLAG: --qos-reserved="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051115 7074 flags.go:64] FLAG: --read-only-port="10255" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051118 7074 flags.go:64] FLAG: --register-node="true" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051121 7074 flags.go:64] FLAG: --register-with-taints="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051125 7074 flags.go:64] FLAG: --registry-burst="10" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051128 7074 flags.go:64] FLAG: --registry-qps="5" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051131 7074 flags.go:64] FLAG: --reserved-cpus="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051134 7074 flags.go:64] FLAG: --reserved-memory="" Oct 31 22:50:22.059141 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051137 7074 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051140 7074 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051143 7074 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051147 7074 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051150 7074 flags.go:64] FLAG: --runonce="false" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051153 7074 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051156 7074 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051159 7074 flags.go:64] FLAG: --seccomp-default="false" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051162 7074 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051165 7074 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051168 7074 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051171 7074 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051174 7074 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051177 7074 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051181 7074 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051185 7074 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051188 7074 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051191 7074 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051194 7074 flags.go:64] FLAG: --system-cgroups="" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051199 7074 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051204 7074 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051207 7074 flags.go:64] FLAG: --tls-cert-file="" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051209 7074 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051214 7074 flags.go:64] FLAG: --tls-min-version="" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051217 7074 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 22:50:22.059870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051220 7074 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051223 7074 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051226 7074 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051229 7074 flags.go:64] FLAG: --v="2" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051233 7074 flags.go:64] FLAG: --version="false" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051241 7074 flags.go:64] FLAG: --vmodule="" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051245 7074 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051248 7074 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051377 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051381 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051384 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051387 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051390 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051393 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051395 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051398 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051400 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051403 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051405 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051408 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051410 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.060503 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051413 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051417 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051420 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051422 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051425 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051429 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051432 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051434 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051437 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051440 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051442 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051445 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051448 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051451 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051453 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051456 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051459 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051461 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051464 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051466 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.061101 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051469 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051472 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051474 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051484 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051487 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051490 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051492 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051495 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051505 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051508 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051510 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051513 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051516 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051519 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051522 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051524 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051527 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051531 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051533 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.061687 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051536 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051538 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051541 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051544 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051546 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051549 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051551 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051554 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051557 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051559 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051561 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051564 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051567 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051569 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051572 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051575 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051577 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051586 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051589 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051591 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.062388 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051594 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051596 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051599 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051602 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051604 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051607 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051609 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051612 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051614 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051617 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051621 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051624 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051628 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051632 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051634 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051637 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051640 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051642 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051645 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.062892 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051647 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051650 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051653 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051655 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051657 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051660 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051663 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051665 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051668 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051670 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051672 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051680 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051685 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051688 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051692 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.051719 7074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:50:22.063379 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051752 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051756 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051759 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051762 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051765 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051768 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051770 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051774 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051777 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051780 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051782 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051785 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051795 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051798 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051801 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051804 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051806 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051809 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051811 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051814 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.063821 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051816 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051819 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051821 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051824 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051827 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051829 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051832 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051835 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051838 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051840 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051843 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051846 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051848 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051851 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051854 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051856 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051859 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051861 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051864 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051868 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.064567 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051871 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051874 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051876 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051879 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051882 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051884 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051887 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051890 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051892 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051894 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051897 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051900 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051905 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051908 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051911 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051914 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051917 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051919 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051922 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.065203 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051924 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051927 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051936 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051939 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051941 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051944 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051946 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051949 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051951 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051954 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051956 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051959 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051967 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051970 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051974 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051977 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051980 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.051983 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052006 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.066022 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052009 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052012 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052014 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052017 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052020 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052022 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052027 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052029 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052032 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052035 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052037 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052040 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052043 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052045 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052048 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052051 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052059 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052062 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052065 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052067 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.066822 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052071 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052074 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052076 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052079 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052082 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052085 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052087 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.052090 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.055690 7074 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.055706 7074 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.055722 7074 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.055727 7074 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055783 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055788 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055791 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055794 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055797 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055800 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055803 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055806 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.067693 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055809 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055812 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055815 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055819 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055823 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055826 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055830 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055833 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055837 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055840 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055842 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055845 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055848 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055851 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055854 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055856 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055859 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055862 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055865 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055868 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.068570 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055871 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055874 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055877 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055880 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055883 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055886 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055889 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055892 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055894 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055897 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055899 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055902 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055905 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055908 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055911 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055913 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055916 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055918 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055921 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.069434 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055923 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055926 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055928 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055931 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055933 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055936 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055938 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055941 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055944 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055946 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055950 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055953 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055956 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055958 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055961 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055964 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055967 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055969 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055972 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055974 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.070246 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055977 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055979 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055982 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.055984 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056002 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056005 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056008 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056011 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056014 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056016 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056019 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056021 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056024 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056027 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056030 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056032 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056035 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056038 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056040 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.071286 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056043 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056045 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056047 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056051 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056053 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056057 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056062 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056066 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056069 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056072 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056075 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056077 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056080 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056082 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056085 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056088 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056090 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056093 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056095 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.072130 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056098 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.056126 7074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056265 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056269 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056272 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056275 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056278 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056280 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056283 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056285 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056288 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056290 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056293 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056295 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056298 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056300 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.073073 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056303 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056306 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056309 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056312 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056315 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056318 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056321 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056323 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056326 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056328 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056331 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056334 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056336 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056339 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056341 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056344 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056347 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056349 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056352 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056354 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.073787 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056357 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056359 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056362 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056364 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056367 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056369 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056372 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056374 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056377 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056379 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056382 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056386 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056391 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056394 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056396 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056399 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056403 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056405 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056408 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.074665 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056410 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056413 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056415 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056418 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056420 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056423 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056430 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056433 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056435 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056438 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056440 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056443 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056446 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056448 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056451 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056453 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056456 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056459 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056461 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056464 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.075518 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056467 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056470 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056474 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056477 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056480 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056484 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056487 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056490 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056493 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056496 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056499 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056502 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056504 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056507 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056509 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056512 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056514 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056517 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056521 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.076623 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056523 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056526 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056529 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056532 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056535 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056538 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056540 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056543 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056545 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056548 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056551 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056554 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056558 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056561 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.056583 7074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056615 7074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:50:22.077581 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056619 7074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056622 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056626 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056629 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056631 7074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056634 7074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056636 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056639 7074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056642 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056645 7074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056648 7074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056651 7074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056653 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056656 7074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056658 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056661 7074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056665 7074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056668 7074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056670 7074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056673 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:50:22.078369 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056676 7074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056678 7074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056681 7074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056684 7074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056686 7074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056690 7074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056692 7074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056695 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056699 7074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056702 7074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056705 7074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056707 7074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056710 7074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056712 7074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056715 7074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056718 7074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056720 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056723 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056725 7074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056728 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:50:22.079400 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056730 7074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056734 7074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056736 7074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056739 7074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056741 7074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056744 7074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056746 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056751 7074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056757 7074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056760 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056763 7074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056765 7074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056768 7074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056771 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056773 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056776 7074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056778 7074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056781 7074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056784 7074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:50:22.080339 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056786 7074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056791 7074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056794 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056796 7074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056799 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056801 7074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056804 7074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056806 7074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056809 7074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056811 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056814 7074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056817 7074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056819 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056822 7074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056824 7074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056827 7074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056830 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056833 7074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056837 7074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:50:22.081197 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056841 7074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056844 7074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056847 7074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056850 7074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056853 7074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056855 7074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056858 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056860 7074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056863 7074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056866 7074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056868 7074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056871 7074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056873 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056876 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056882 7074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056884 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056887 7074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056890 7074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056892 7074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:50:22.081852 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056895 7074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056897 7074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056900 7074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056903 7074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056906 7074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056908 7074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056911 7074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:22.056913 7074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.056972 7074 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.058662 7074 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.058744 7074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.059238 7074 server.go:1014] "Starting client certificate rotation" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.059369 7074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.059413 7074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:44:11 +0000 UTC" deadline="2028-06-12 17:48:16.973002293 +0000 UTC" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.059438 7074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14154h57m54.913565664s" Oct 31 22:50:22.082346 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.059679 7074 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:50:22.082751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.060733 7074 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:50:22.082751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.061820 7074 log.go:25] "Validated CRI v1 runtime API" Oct 31 22:50:22.082751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.063833 7074 log.go:25] "Validated CRI v1 image API" Oct 31 22:50:22.082751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.064556 7074 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 22:50:22.082751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.068099 7074 fs.go:135] Filesystem UUIDs: map[12d4ebba-a539-48c7-90c6-c7dc3ea13125:/dev/nvme0n1p3 43995d6f-7e30-4c43-ad9a-c5467c27c251:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 22:50:22.082913 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.068127 7074 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6/userdata/shm major:0 minor:454 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476/userdata/shm major:0 minor:502 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bc02dd39b5e5ba8babd046b0dc520e6a6d5548ce9e691a23a1e6b71cd4e4561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bc02dd39b5e5ba8babd046b0dc520e6a6d5548ce9e691a23a1e6b71cd4e4561/userdata/shm major:0 minor:505 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43e9514ffbd7f79e2a39e3a3aa62a935084a55756861a4deb1ad0a9759631cbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43e9514ffbd7f79e2a39e3a3aa62a935084a55756861a4deb1ad0a9759631cbb/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eb8fac7d20461bbe7ab8068c6039524c9b69568b5aeec556efdbf38e0608232/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eb8fac7d20461bbe7ab8068c6039524c9b69568b5aeec556efdbf38e0608232/userdata/shm major:0 minor:504 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~projected/kube-api-access-8kbn7:{mountpoint:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~projected/kube-api-access-8kbn7 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/volumes/kubernetes.io~projected/kube-api-access-mzddt:{mountpoint:/var/lib/kubelet/pods/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/volumes/kubernetes.io~projected/kube-api-access-mzddt major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~projected/kube-api-access-grpck:{mountpoint:/var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~projected/kube-api-access-grpck major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~projected/kube-api-access-qwxxz:{mountpoint:/var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~projected/kube-api-access-qwxxz major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~projected/kube-api-access-jvh7r:{mountpoint:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~projected/kube-api-access-jvh7r major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39248804-861a-4442-a717-66a2467e2c72/volumes/kubernetes.io~projected/kube-api-access-qhwqt:{mountpoint:/var/lib/kubelet/pods/39248804-861a-4442-a717-66a2467e2c72/volumes/kubernetes.io~projected/kube-api-access-qhwqt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/volumes/kubernetes.io~projected/kube-api-access-dkvb2:{mountpoint:/var/lib/kubelet/pods/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/volumes/kubernetes.io~projected/kube-api-access-dkvb2 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48ac9bdf-c637-46e4-9fac-53c43f3fd161/volumes/kubernetes.io~projected/kube-api-access-gfxx2:{mountpoint:/var/lib/kubelet/pods/48ac9bdf-c637-46e4-9fac-53c43f3fd161/volumes/kubernetes.io~projected/kube-api-access-gfxx2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56499b86-c2fe-429c-88b8-cc5be0e63840/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56499b86-c2fe-429c-88b8-cc5be0e63840/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59b2c63f-8550-4152-9759-d522b6b4e520/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/59b2c63f-8550-4152-9759-d522b6b4e520/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/config-reloader/2 major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/prometheus/15 major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~empty-dir/config-out major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/kube-api-access-ddkpr:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/kube-api-access-ddkpr major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/tls-assets major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/config major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/web-config major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~projected/kube-api-access-n47md:{mountpoint:/var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~projected/kube-api-access-n47md major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~secret/metrics-certs major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~projected/kube-api-access-bp8ps:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~projected/kube-api-access-bp8ps major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98b6f6c7-3878-491d-8a05-131c56390f37/volumes/kubernetes.io~projected/kube-api-access-gmgqd:{mountpoint:/var/lib/kubelet/pods/98b6f6c7-3878-491d-8a05-131c56390f37/volumes/kubernetes.io~projected/kube-api-access-gmgqd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~projected/kube-api-access-qv2q2:{mountpoint:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~projected/kube-api-access-qv2q2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~projected/kube-api-access-5fcqm:{mountpoint:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~projected/kube-api-access-5fcqm major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~projected/kube-api-access-lzz45:{mountpoint:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~projected/kube-api-access-lzz45 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~projected/kube-api-access-f4sl8:{mountpoint:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~projected/kube-api-access-f4sl8 major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~projected/kube-api-access-b9cf2:{mountpoint:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~projected/kube-api-access-b9cf2 major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~projected/kube-api-access-x6jwb:{mountpoint:/var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~projected/kube-api-access-x6jwb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/alertmanager/9 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/config-reloader/6 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~empty-dir/config-out major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/kube-api-access-cpmpb:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/kube-api-access-cpmpb major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/tls-assets major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/config-volume major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/web-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e383e48a-7ee2-4c25-a5e4-d708af31e31b/volumes/kubernetes.io~projected/kube-api-access-dzzxd:{mountpoint:/var/lib/kubelet/pods/e383e48a-7ee2-4c25-a5e4-d708af31e31b/volumes/kubernetes.io~projected/kube-api-access-dzzxd major:0 minor:501 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~projected/kube-api-access-z54qb:{mountpoint:/var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~projected/kube-api-access-z54qb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b13f57-54cc-4499-a9a5-44eec96c4488/volumes/kubernetes.io~projected/kube-api-access-lrdtc:{mountpoint:/var/lib/kubelet/pods/f6b13f57-54cc-4499-a9a5-44eec96c4488/volumes/kubernetes.io~projected/kube-api-access-lrdtc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~projected/kube-api-access-j9rgk:{mountpoint:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~projected/kube-api-access-j9rgk major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f2ef11a39407ce12cb52a1c1eef6c150190ed33b1563c26b7f071fe77648d498/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4f6ae8b030d721ae0a12e606589d8d46b63cc5ce8cd25303997f7ba3dbb9da0e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/642013267b34433a66bf4c76811f1ef1a9631c40eab2f0f528496a3a221041a2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ba394310ad876187e964cedd538e8b4163d05ac04d37834855c0dccd266f92bd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4717d9c46dd11adfee450e9f553dd96d8d12127aa556e25d2327abd1db7db5d8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d62e69b7617ea28e56a2610e1f917a62094f16830cb28743f96bea2f0ce5ed1a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a91e43f2c3fdaffe9fbef25dfcd6a841c372f563657137e4fdcb2241b25b1afd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/014a917440cbcbb4f4761c5b4a0cdc4ecb7f36f8ffa141ac2d769c6baffe0850/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/38735c85d3e7b9cb46818cfcd35b09a6e391fed8e0edaf722fff7ae83ff9b468/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/b27c47388fd75a1f35b3c709d038b6737a36eae730e7eaf1d434bf59a3f17b04/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8d66494ce380841d890f01b2bdbf473f2ee678331eb99f880c1fd4c25450784c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/dc6c569f7edf99c5695a9dce27279889130d2bce024ca939d9c22af3c655ebd3/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f434f352df943e0da27853d05b9be2a21faff1003af9c37dcb874a73577e9a16/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/6aab3a419f9764260b38b6121e73b41921b68662c2b7bdd2bf7779d1e8b2f699/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/56c93445874c7ebd086bd717e3e3ee50064738379a1f8fc7d0984a7884ce79eb/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/c78dfe4e5a87a2df082614497f5a8c1b0f47e88ae750c414e5108ac20cc9f468/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/ef0ef58961051bb1963cc6a1166dea24a16417193c8647e2a9670797c3b24c4f/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/3174f58c080c671c63dff77da9afd83ddb3c657457b75951c35b2effccd38c8f/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7394d05a1608bc14bff801def567f791841acae07b5b46b0d11bb0a3fde8229a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4d96416d376fcc1941474921e90fc1cd48098e108dd1c8596e1f19e1d094eee9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/01d424c61fe04048e52a1748389a08de838562869c72c328230ee52e24f16bd1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/47b8dcd340dfb553fd082d3e3ba333f3e318037a1342767d1734cf2344e87104/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4f5b99a5b107a5d5c4de803ffabfc26628f4de7b210aeccb69b2608cb109bf2d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/dfb47123704fc57a160f02b40fdd48478dde508afeaeba6d76de38589996c420/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1195cf6553104e98d34285fc02c48f6a822c12a6175f4bd7c8eccd415da1fbf5/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b81c783988eb6522b77420b0e9daa3be0de9a831db7497619df1eae7d7d115ab/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/cd689e53f7808832cf48ef5c60bc5eebd1fe8c2056b0f5fd6adc3443bf75a6a9/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/49a2ceca61897faec79019c5ededce66d37b7ce4d05d2340d2f0970bc9388993/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f76bf9083ca6da273a25dc382b5c229efacfa240f5ea6f967a0f0ecbb5659af3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/ccbb0eb57f2144d8dd80136cef93dd25fee6fb3285796cb4e0905acf904f5a88/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/a39c9fc58d7dde515d2c0367ffb087f082292b1984b7489f16e52ee618270584/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2a7e858a6a87d5e69c7a279f6f7154b27e286e6cc1446100ab950d2612a11cd7/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/ef74ceaf1a36a1f0fc839bae2232f6c76b56a8936e10895088607cb2137e3a15/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/f227d865c4add38f71ac495beda48ec02ef16e687a3cf253e6f8102f81978147/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/25bab08d2c504748270c6655a544371549c49ed1fe5f03e74f62790cd9077840/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/c54cda17a9eaab3fdee54c4e893381cfe2c0d9dbe788ba6a34a1724b2e362764/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/7f0e5906f51f48ac2b23d6fed3a3b04899cfacb0d483560ae24b223f01a3190f/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/1d70082ba8b2137f76b45033fa15d59b55a99d0910f8533af9c681a47d69dc2e/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/941da0107ab3f5fcfe6d560fd597acb125bb273a5b066a6ebba16cf6bfbb4c65/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/84b680a380ac8149872cf5580702415e8db0ba309425a402ffac93c550543d5e/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/7cebbb60108ce7b1b1272a88b7be5cd60cccbf0acd94c1bb5885fdf0e03cf1a2/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/c7cd870de1cd341086f63ab98d59580802b55e33108e48cce506e7eb02c471cb/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/6803fff4a0f76b3ec15ca5de7d78b91803d4dceafb5ba1267efec978d0226172/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/89e9341510f25bcbff3f2e1513fbd6e395ff2e4a8e3ca88580462becda89cac8/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/0cbde607c625ff5fb742ede15b825ce28c4f635728ca48f1f3e1184423aea3a5/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/0a8b73de9b522007b7f05e2baa07e39526f20947fc6e466406eec21ed753f262/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/87927d56a88e357e5d3da19e45a87936111cfba30bb4b8ebf609e7250f2a6074/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/150aa0137558d171d9121256e475ab4357a81503d7ee3ccc054adc3d49a9e1c2/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/c71011ebd8cb0e3a58e8c2bb75a6e5628f2c4490ff81ab8201bd20bb4a098bf5/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/4b9f8080e8436912e231375478c8635c3286a63fc25d0b2abed4ecfba82502d5/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/a509c0fdb01665b9445a9b14287abffd9d4d4e558b6b746b94c8cffc6a60db84/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/b551677dbcdcbd26079b7f293cacba2fdacfe0bbcd72b9a0ee7560b5ced7f8b4/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/e133bc0c2e81678f50720902a3145b0c5e99ffd6c37d33235d0a9788761e9ceb/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/5fa47c03d038ce12f725c941b13cc4b807f201aafaa4d9feee78ce1f33583a9a/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/495f3bc0961bbff952846d62deaabcb589533e1c13a87be4394df2b07105c759/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/abba869f23999c45f39c32b5a422c98507d4f76321e441ebff82e5731a0a2953/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/f50cbc467e662e0e13adc241c6baa6af2a31b01323ecaea399e708b1eab40e77/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/2a715ac86d248721fe27dad81acb801fb06f3c624e9ffbd366afda601ee598d7/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/7356c8b3199e120cdf4c43a4f685c530d7cf5663cd7e2924aaec40e0463c0b0e/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/f6694984b770596619ac421c646143ce9da3e99cafcd73118506564a7059319c/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/e972d90e57b62d23a7f6f01647c16f4433cd6f50d9959c31216cc92d3f7fc4ec/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/323c9ff915767cdd2fd2b591232c92256e97ee9f68e014aa3ad6a5e1f9390193/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/f033cdc7f0fffe336e1e3f7857372d4e5d6ab5d7efa1e7207d3ace43652b8b05/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/d10273991b504abf86286d43ef27a36c4fff6dfd5a0e66a2c73a6994e97d1265/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/0634921c7e61c71530cc619eb51bbc7be911d96ed29717ea0707c309a456a71c/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/699274bbbfc139128c96c6b12cf86cf422e1f8f6a5fafbdf8453f4b7b0016561/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/d907cc61188ba41cd7f269dd8512e665c701c50f4c178e95ad07586fc224ba15/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/db289a89db22686cb224e4539b8ca2a888af588eab9042c7e4b8a2b2605c6a57/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/15ff33d24fba277bc923fc22ba8c1c56870eb07eba3162da5f85d3f114404659/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/a437ba4cea1d4e288c502debe3e54f6fac589f51e550ae707871ab85113f2bd1/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/7184d5db024ea5a7e62f7af0ce7071de77e7be12def9f7902d295f77de20f09b/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/c790e969beb25674e5c85cd74b5d094dd23651f156e90f1a70efb30850a715d4/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/1b233d910d28c30983783034d981adc75ac9985466fcd9e253bc5915943b98ab/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/147e0c3b0c0c67354df7433ab91a7577383ac961bba1aa30803b1eb78b40123d/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/46ce8e31f97279a40edddddcf0b0f745c7f8c611b85d9e8d27067ed44394bfca/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/a7afd0fdfde7e84b0f87f807c1e23938673c2af4275e5fc93c30d358dc8f185b/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/92050c1fb24e0f6684feafee55b2e07d1301f659c951dc6a2118f8ffcb7dc095/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/1a4cf7d7af60dd3d109dbf347fdbd85d2d2421441575833e5f88e03aba979448/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/1ba693c2d53e78df139d5cc86ab3af988a2fcf1b12bcc6c40dff212f1b7a023b/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/68679cd7fe02893f46b8852b0eae9ca1e6068334d54387b19ea2a8c8c607ea31/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b8b95c4d108a26d5f50d27967849a098c5d627831334dc15a4ce492196e7a3dc/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/78b6c1fa0eb9216e8cfdae6d6d2f5097ccdcf6a99a5026989c0dfa0ec960a6d2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4beb373f9daacc116e129013090699083dccd86a811eb288da1c2eda139a3c98/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/45c1db73218bda52ddc2ba98df4179f6b48094e277dfdf6f4e078df82ea832c6/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f87436c14e986213a234dec75127da6ad3bdff2473f684888bd86b25d4189c6f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0213b5e13c289bbff472a022caa14755766a4ba5204966c6f22653fac2adaaa8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/010515edd5d02d535938c3d15705858981e2a8c5f3ec62e673265de92d9c9b29/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.081275 7074 manager.go:217] Machine: {Timestamp:2026-10-31 22:50:22.080223078 +0000 UTC m=+0.062965449 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102289 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db56fe036d80fd3eb8c89b929d11c SystemUUID:ec2db56f-e036-d80f-d3eb-8c89b929d11c BootID:da1c4571-b4cf-4d1c-9c98-2095b93d800a Filesystems:[{Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~projected/kube-api-access-bp8ps DeviceMajor:0 DeviceMinor:312 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2742ff7cee15e8e68339108a65b206ef6c451e689a25091b1ddddbd878aa3f6b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ba645045038b56f84100827820abba342bf2b62ccbb15beb724d43553d6a0cf/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:307 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:309 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec1fc99d62f9be4613e094d2347167c097198d472cdee49ed24a208d3d591fea/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43e9514ffbd7f79e2a39e3a3aa62a935084a55756861a4deb1ad0a9759631cbb/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:295 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/946921c9a0da09e3dbb72a44c1fe23da045c635fa2d4bc9257f8d86a124703c6/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~projected/kube-api-access-jvh7r DeviceMajor:0 DeviceMinor:265 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6df8c2aea8e9e5bb0c652b913f7a94027445687f9f266fedd967f04e78ce87a/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:342 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:343 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/56499b86-c2fe-429c-88b8-cc5be0e63840/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:500 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~projected/kube-api-access-x6jwb DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:443 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:376 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:446 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~projected/kube-api-access-j9rgk DeviceMajor:0 DeviceMinor:52 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:446 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:447 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:441 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/59b2c63f-8550-4152-9759-d522b6b4e520/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/volumes/kubernetes.io~projected/kube-api-access-mzddt DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~projected/kube-api-access-qwxxz DeviceMajor:0 DeviceMinor:363 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:450 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:241 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:446 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89741e1bc0526cefd3a006440b231e5945c378a57a12d2fbfdd6e5e712bfb63c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/volumes/kubernetes.io~projected/kube-api-access-dkvb2 DeviceMajor:0 DeviceMinor:54 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:258 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:291 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/27c3dce2-82b9-4868-8ec9-2d480ff15a1c/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:362 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:445 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1312fd378b6e40f630165a81e0548ff497ad701aa9d2835349a9b9755b2b5476/userdata/shm DeviceMajor:0 DeviceMinor:502 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de46a038f120aedaa8a1fa6f45a653162ab32e1db56140ad0e880b32d642d62b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~projected/kube-api-access-ddkpr DeviceMajor:0 DeviceMinor:451 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6be61f250a0e611aa9a06522388131d535b046f8f8ad0a53551c427e9ea8766b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:292 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:353 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:442 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46e95aa6689aaa2e9f509086f7698f883c03c1acab149a116bf9a88668d80217/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:437 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6/userdata/shm DeviceMajor:0 DeviceMinor:454 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~projected/kube-api-access-n47md DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~projected/kube-api-access-8kbn7 DeviceMajor:0 DeviceMinor:263 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bc02dd39b5e5ba8babd046b0dc520e6a6d5548ce9e691a23a1e6b71cd4e4561/userdata/shm DeviceMajor:0 DeviceMinor:505 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/48ac9bdf-c637-46e4-9fac-53c43f3fd161/volumes/kubernetes.io~projected/kube-api-access-gfxx2 DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~projected/kube-api-access-b9cf2 DeviceMajor:0 DeviceMinor:246 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:290 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:306 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:292 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e383e48a-7ee2-4c25-a5e4-d708af31e31b/volumes/kubernetes.io~projected/kube-api-access-dzzxd DeviceMajor:0 DeviceMinor:501 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39248804-861a-4442-a717-66a2467e2c72/volumes/kubernetes.io~projected/kube-api-access-qhwqt DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62315ea1e6a22c6efb9da5f1f91f1f14b741fccb482e1abd05fc340f3b26bbae/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:245 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b13f57-54cc-4499-a9a5-44eec96c4488/volumes/kubernetes.io~projected/kube-api-access-lrdtc DeviceMajor:0 DeviceMinor:51 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:310 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:446 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6de4ea6c3cf0b023d00d9b844bd6e3460e5434ab1f25fca36cac894238900f66/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b78be0b3b4d0354a419346c02b7c1ad5f347992d68aa9c6349469288993a388/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:297 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98b6f6c7-3878-491d-8a05-131c56390f37/volumes/kubernetes.io~projected/kube-api-access-gmgqd DeviceMajor:0 DeviceMinor:53 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volumes/kubernetes.io~projected/kube-api-access-lzz45 DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~projected/kube-api-access-cpmpb DeviceMajor:0 DeviceMinor:298 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d187ad0-d0cd-4442-a0b5-ab7e324cacda/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:261 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1152af8aaabe0232e32918e7195d4038ad8ac0be2fcf1bd4e14e05e133cd8dfb/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:380 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:448 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb149490-9112-463a-b5ef-107f84f04d0c/volumes/kubernetes.io~projected/kube-api-access-z54qb DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:262 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ca1b181-263f-43a9-96fb-2a2efbd8c963/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:496 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f91b6ba2-b532-47ef-b1de-ea7a018f2355/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/903ce85a6f96f8a2738e102aaa89517a910b82bfbed7f3dd1b86da26c705cc74/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26fd3b3c-9c2e-43b7-9660-ec1ff811348e/volumes/kubernetes.io~projected/kube-api-access-grpck DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:260 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4576bb907d3ddae37717ce199df0e72efa6a7ae5ced6a8c126e53448e4717871/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:449 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9004f5b-8f45-4dfe-b883-d485d3b43582/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b2252e42-a7dd-4822-83ca-65aff7bb802a/volumes/kubernetes.io~projected/kube-api-access-5fcqm DeviceMajor:0 DeviceMinor:264 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes/kubernetes.io~projected/kube-api-access-qv2q2 DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:444 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa88e10738ecc47c6b963402b39f0577cf8acb8621f395756a8d535608572b49/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9fe7d02-92f2-41fd-9122-abdfdc159006/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eb8fac7d20461bbe7ab8068c6039524c9b69568b5aeec556efdbf38e0608232/userdata/shm DeviceMajor:0 DeviceMinor:504 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:294 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9ed1c1-75ac-42cc-8613-67993c17b9a5/volumes/kubernetes.io~projected/kube-api-access-f4sl8 DeviceMajor:0 DeviceMinor:354 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1107aefd212ccb4 MacAddress:de:ba:ec:4f:16:9e Speed:10000 Mtu:8901} {Name:1152af8aaabe023 MacAddress:8e:61:6b:24:65:5b Speed:10000 Mtu:8901} {Name:1312fd378b6e40f MacAddress:52:12:f9:ec:68:16 Speed:10000 Mtu:8901} {Name:3304a7aa837fd7e MacAddress:32:66:2a:2f:ac:25 Speed:10000 Mtu:8901} {Name:3bc02dd39b5e5ba MacAddress:96:72:e5:a7:81:56 Speed:10000 Mtu:8901} {Name:43e9514ffbd7f79 MacAddress:d6:0c:b8:65:21:a6 Speed:10000 Mtu:8901} {Name:4576bb907d3ddae MacAddress:c2:a3:e4:ca:98:14 Speed:10000 Mtu:8901} {Name:4eb8fac7d20461b MacAddress:9e:8d:0d:73:3a:b9 Speed:10000 Mtu:8901} {Name:62315ea1e6a22c6 MacAddress:c2:67:dd:43:b0:8b Speed:10000 Mtu:8901} {Name:6969d27a33a7e41 MacAddress:a6:94:1b:94:67:2f Speed:10000 Mtu:8901} {Name:946921c9a0da09e MacAddress:06:7c:78:76:1e:ec Speed:10000 Mtu:8901} {Name:9b78be0b3b4d035 MacAddress:9e:ae:23:67:87:c7 Speed:10000 Mtu:8901} {Name:b3a12457099091d MacAddress:ce:7b:e6:a1:44:f2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e6:08:e9:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:c6:0e:96:36:df Speed:0 Mtu:8901} {Name:c6df8c2aea8e9e5 MacAddress:be:d9:2d:c6:11:a7 Speed:10000 Mtu:8901} {Name:de46a038f120aed MacAddress:42:bc:70:07:90:2d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e6:08:e9:55 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:4e:3e:08:4e:db:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:48:97:8f:54:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082191 7074 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082248 7074 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082500 7074 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082519 7074 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-157.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082644 7074 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082653 7074 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082663 7074 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082682 7074 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082699 7074 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082843 7074 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082946 7074 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.082978 7074 kubelet.go:493] "Attempting to sync node with API server" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083002 7074 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083012 7074 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083021 7074 kubelet.go:405] "Adding apiserver pod source" Oct 31 22:50:22.083289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083048 7074 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 22:50:22.083931 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083819 7074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 22:50:22.083931 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.083832 7074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:50:22.084375 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.084356 7074 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 22:50:22.087836 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.087821 7074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 22:50:22.087836 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.087834 7074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:50:22.087946 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.087936 7074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 22:50:22.088152 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088139 7074 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 22:50:22.088190 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088180 7074 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 22:50:22.088298 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088287 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088303 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088309 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088317 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088327 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088334 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088339 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088345 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088352 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 22:50:22.088358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088358 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 22:50:22.088705 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088366 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 22:50:22.088705 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088378 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 22:50:22.088705 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088417 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 22:50:22.088705 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.088427 7074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 22:50:22.091867 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.091852 7074 server.go:1263] "Started kubelet" Oct 31 22:50:22.091979 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.091927 7074 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 22:50:22.092054 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.091972 7074 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 22:50:22.092106 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.092075 7074 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 22:50:22.092395 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.092377 7074 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 22:50:22.092401 ip-10-0-131-157 systemd[1]: Started Kubernetes Kubelet. Oct 31 22:50:22.093535 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.093517 7074 server.go:317] "Adding debug handlers to kubelet server" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098109 7074 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098142 7074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098200 7074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:44:12 +0000 UTC" deadline="2028-07-13 23:49:57.812786814 +0000 UTC" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098221 7074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14904h59m35.714569228s" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098245 7074 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.098253 7074 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-157.us-east-2.compute.internal\" not found" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098258 7074 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 22:50:22.099019 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.098269 7074 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 22:50:22.099732 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.099709 7074 factory.go:55] Registering systemd factory Oct 31 22:50:22.099732 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.099734 7074 factory.go:223] Registration of the systemd container factory successfully Oct 31 22:50:22.100118 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.099981 7074 factory.go:153] Registering CRI-O factory Oct 31 22:50:22.100118 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.100119 7074 factory.go:223] Registration of the crio container factory successfully Oct 31 22:50:22.100273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.100175 7074 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 22:50:22.100273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.100193 7074 factory.go:103] Registering Raw factory Oct 31 22:50:22.100273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.100208 7074 manager.go:1199] Started watching for new ooms in manager Oct 31 22:50:22.100541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.100526 7074 manager.go:319] Starting recovery of all containers Oct 31 22:50:22.101769 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101729 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101769 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101784 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101798 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101811 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101824 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 22:50:22.101855 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101836 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101865 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39248804-861a-4442-a717-66a2467e2c72" volumeName="kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101883 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101897 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b6f6c7-3878-491d-8a05-131c56390f37" volumeName="kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101909 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101922 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101935 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101947 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101962 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.101977 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b6f6c7-3878-491d-8a05-131c56390f37" volumeName="kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102005 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9fe7d02-92f2-41fd-9122-abdfdc159006" volumeName="kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102037 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102050 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102062 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102075 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102089 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102101 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102129 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102142 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fd3b3c-9c2e-43b7-9660-ec1ff811348e" volumeName="kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102155 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27c3dce2-82b9-4868-8ec9-2d480ff15a1c" volumeName="kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102168 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102181 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102200 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102213 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102225 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fd3b3c-9c2e-43b7-9660-ec1ff811348e" volumeName="kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert" seLinuxMountContext="" Oct 31 22:50:22.102226 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102238 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59b2c63f-8550-4152-9759-d522b6b4e520" volumeName="kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102252 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102266 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b13f57-54cc-4499-a9a5-44eec96c4488" volumeName="kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102278 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102289 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102302 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102314 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102326 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102339 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102352 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb149490-9112-463a-b5ef-107f84f04d0c" volumeName="kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102367 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102380 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ca1b181-263f-43a9-96fb-2a2efbd8c963" volumeName="kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102392 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102404 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102434 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102447 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9fe7d02-92f2-41fd-9122-abdfdc159006" volumeName="kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102460 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9fe7d02-92f2-41fd-9122-abdfdc159006" volumeName="kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102474 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102488 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102503 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102517 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102531 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59b2c63f-8550-4152-9759-d522b6b4e520" volumeName="kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102546 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102561 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102577 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39248804-861a-4442-a717-66a2467e2c72" volumeName="kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102603 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56499b86-c2fe-429c-88b8-cc5be0e63840" volumeName="kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102618 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102631 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9fe7d02-92f2-41fd-9122-abdfdc159006" volumeName="kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102643 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102656 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102669 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102682 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102702 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9004f5b-8f45-4dfe-b883-d485d3b43582" volumeName="kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102715 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102729 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ef2f42-14fa-4d62-be14-ef9fe9fd92f5" volumeName="kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102742 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102757 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102769 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c47d3f0-fedf-456a-b4db-dc5115e9c5d6" volumeName="kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102785 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102797 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102810 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ca1b181-263f-43a9-96fb-2a2efbd8c963" volumeName="kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102823 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102837 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102851 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b13f57-54cc-4499-a9a5-44eec96c4488" volumeName="kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102865 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27c3dce2-82b9-4868-8ec9-2d480ff15a1c" volumeName="kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102879 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102892 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102904 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102916 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9fe7d02-92f2-41fd-9122-abdfdc159006" volumeName="kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102930 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102944 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102957 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d" volumeName="kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.102971 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103003 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103017 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103030 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f91b6ba2-b532-47ef-b1de-ea7a018f2355" volumeName="kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103045 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f91b6ba2-b532-47ef-b1de-ea7a018f2355" volumeName="kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103058 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103071 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb149490-9112-463a-b5ef-107f84f04d0c" volumeName="kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103083 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103095 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b6f6c7-3878-491d-8a05-131c56390f37" volumeName="kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103110 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103124 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" volumeName="kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103137 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb149490-9112-463a-b5ef-107f84f04d0c" volumeName="kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103150 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb149490-9112-463a-b5ef-107f84f04d0c" volumeName="kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103161 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103174 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103186 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b6f6c7-3878-491d-8a05-131c56390f37" volumeName="kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103201 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9004f5b-8f45-4dfe-b883-d485d3b43582" volumeName="kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103214 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" volumeName="kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103226 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9004f5b-8f45-4dfe-b883-d485d3b43582" volumeName="kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103240 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b13f57-54cc-4499-a9a5-44eec96c4488" volumeName="kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103252 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48ac9bdf-c637-46e4-9fac-53c43f3fd161" volumeName="kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103264 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103277 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ec9d28-478d-45f2-82f5-8197104db046" volumeName="kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103290 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103303 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9004f5b-8f45-4dfe-b883-d485d3b43582" volumeName="kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103316 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103328 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c47d3f0-fedf-456a-b4db-dc5115e9c5d6" volumeName="kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103342 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c47d3f0-fedf-456a-b4db-dc5115e9c5d6" volumeName="kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103355 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2252e42-a7dd-4822-83ca-65aff7bb802a" volumeName="kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103386 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f91b6ba2-b532-47ef-b1de-ea7a018f2355" volumeName="kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103400 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c47d3f0-fedf-456a-b4db-dc5115e9c5d6" volumeName="kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103413 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d" volumeName="kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103425 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a737f063-f6c6-4120-bd76-2b666dd9b154" volumeName="kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103439 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e383e48a-7ee2-4c25-a5e4-d708af31e31b" volumeName="kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103459 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f" volumeName="kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103472 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ef2f42-14fa-4d62-be14-ef9fe9fd92f5" volumeName="kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103484 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9ed1c1-75ac-42cc-8613-67993c17b9a5" volumeName="kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103503 7074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" volumeName="kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume" seLinuxMountContext="" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103516 7074 reconstruct.go:97] "Volume reconstruction finished" Oct 31 22:50:22.103460 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.103525 7074 reconciler.go:29] "Reconciler: start to sync state" Oct 31 22:50:22.107716 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.105679 7074 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:50:22.113186 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.113159 7074 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:50:22.114524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.114502 7074 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 22:50:22.116118 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.116096 7074 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:50:22.132337 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.132288 7074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 22:50:22.134035 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.134010 7074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 22:50:22.134035 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.134036 7074 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 22:50:22.134229 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.134064 7074 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 22:50:22.134229 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.134106 7074 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 22:50:22.139165 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.139138 7074 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:50:22.143617 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.143581 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:50:22.143745 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.143702 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:50:22.146652 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.146617 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f" exitCode=0 Oct 31 22:50:22.160450 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.160426 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c" exitCode=0 Oct 31 22:50:22.166413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166386 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="f4a4eeaffd81bb3bb4d0eb980b83107da7c42253b77be43f2e12bfa863a7e18c" exitCode=0 Oct 31 22:50:22.166413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166405 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="a047ca7df029db8d90aad97d0975de39a7085a704c26306d976013904a97ed14" exitCode=0 Oct 31 22:50:22.166547 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166417 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="63355b47bc1c40fc865fb4bd14c01650910aa0da64edc991f650b80d3ba72c0c" exitCode=0 Oct 31 22:50:22.166547 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166426 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="b7f51b9a6c907d1d9c069cbb4b1851a9ad52d553573c1a289703eedf1e5be41e" exitCode=0 Oct 31 22:50:22.166547 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166431 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="50a20a5596d891021ceb838721b43a37aa1c3ec236a2bdd58df13aba27dea98b" exitCode=0 Oct 31 22:50:22.166547 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.166436 7074 generic.go:356] "Generic (PLEG): container finished" podID="98b6f6c7-3878-491d-8a05-131c56390f37" containerID="487865b2eccb4a82bc27086d98d762941bd26843ce5c747dfdba4978e35b129a" exitCode=0 Oct 31 22:50:22.183240 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.183221 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:50:22.183564 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.183540 7074 generic.go:356] "Generic (PLEG): container finished" podID="c9fe7d02-92f2-41fd-9122-abdfdc159006" containerID="7f9b300ed9a67813c080942d4a28d0d4a0048cff0ddf9b41f6c28e0e96629014" exitCode=1 Oct 31 22:50:22.189576 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.189556 7074 generic.go:356] "Generic (PLEG): container finished" podID="0d187ad0-d0cd-4442-a0b5-ab7e324cacda" containerID="a52fd81782f8af33190d392014919f893337bf29d8f413465f098a0eba6dd37e" exitCode=0 Oct 31 22:50:22.200419 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.200398 7074 generic.go:356] "Generic (PLEG): container finished" podID="8a68def89a8e2f4b1d88fecf85bfdb5e" containerID="bc707c338344f8d3decfb90232747a175724a8d1800f58b693eb6f3fedfd5880" exitCode=0 Oct 31 22:50:22.227282 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.227215 7074 manager.go:324] Recovery completed Oct 31 22:50:22.234228 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.234201 7074 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 22:50:22.255586 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255559 7074 cpu_manager.go:226] "Starting" policy="none" Oct 31 22:50:22.255586 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255579 7074 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 22:50:22.255798 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255604 7074 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 22:50:22.255798 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255772 7074 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255789 7074 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255815 7074 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255824 7074 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255832 7074 policy_none.go:50] "Start" Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255844 7074 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 22:50:22.255958 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.255858 7074 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 22:50:22.257462 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.257444 7074 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 22:50:22.257530 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.257466 7074 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 22:50:22.257530 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.257477 7074 policy_none.go:44] "Start" Oct 31 22:50:22.262232 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.262204 7074 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 22:50:22.264745 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.264732 7074 manager.go:341] "Starting Device Plugin manager" Oct 31 22:50:22.265306 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.264758 7074 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 22:50:22.265306 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265307 7074 server.go:87] "Starting device plugin registration server" Oct 31 22:50:22.265563 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265552 7074 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 22:50:22.265596 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265564 7074 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 22:50:22.265659 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265638 7074 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 22:50:22.265757 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265725 7074 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 22:50:22.265757 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265756 7074 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 22:50:22.265851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265761 7074 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 22:50:22.266088 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.265812 7074 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:50:22.265748566Z","UUID":"89accd58-f9e0-4137-bc07-6a7a0d5258f4","Handler":null,"Name":"","Endpoint":""} Oct 31 22:50:22.267357 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.267333 7074 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 22:50:22.267429 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.267375 7074 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 22:50:22.267604 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.267583 7074 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 22:50:22.366529 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.366485 7074 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:50:22.368606 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.368582 7074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:50:22.368715 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.368612 7074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:50:22.368715 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.368624 7074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-157.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:50:22.368715 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.368673 7074 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.381436 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.381416 7074 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.381514 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.381504 7074 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.434591 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.434519 7074 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal"] Oct 31 22:50:22.435323 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435296 7074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.435426 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435386 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"6e81279d3f58a67f979dda36c603d642af1d0dc7fa186d1a7b6c63be97d7c7b5"} Oct 31 22:50:22.435463 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435439 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerDied","Data":"bc707c338344f8d3decfb90232747a175724a8d1800f58b693eb6f3fedfd5880"} Oct 31 22:50:22.435494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435459 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"8a68def89a8e2f4b1d88fecf85bfdb5e","Type":"ContainerStarted","Data":"0d027c2c2dcc164d51d95d7477b416d283deafe0774f719a2e4302319e022eca"} Oct 31 22:50:22.435494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435477 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"6d4ec94d4947568ba04ce8da2a0fa5050782627fc7eccfa40a3c733897aec838"} Oct 31 22:50:22.435574 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435492 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" event={"ID":"72de75e4a93b82a0133be1d6ed9e0a51","Type":"ContainerStarted","Data":"187aa92f3ac806bac29ab67912b86230bb5f22517d6468f1756127ea06c38506"} Oct 31 22:50:22.435574 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.435504 7074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.449824 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.449799 7074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:50:22.449932 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.449850 7074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.450301 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.450285 7074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:50:22.450343 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:22.450330 7074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.516669 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.516624 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.516669 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.516667 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.516853 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.516692 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617247 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617291 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617292 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617496 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617310 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617496 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617355 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617496 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617365 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72de75e4a93b82a0133be1d6ed9e0a51-config\") pod \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"72de75e4a93b82a0133be1d6ed9e0a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:22.617496 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:22.617366 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a68def89a8e2f4b1d88fecf85bfdb5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" (UID: \"8a68def89a8e2f4b1d88fecf85bfdb5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:23.083910 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.083872 7074 apiserver.go:52] "Watching apiserver" Oct 31 22:50:23.108651 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.108623 7074 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 22:50:23.109340 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.109314 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qqjd2","kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-t48z9","openshift-insights/insights-runtime-extractor-j2gk2","openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc","openshift-monitoring/node-exporter-7wpwb","openshift-monitoring/thanos-querier-5896fc6d85-lpp69","openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p","openshift-multus/multus-additional-cni-plugins-ctfvq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk","openshift-multus/multus-rdsnf","openshift-ovn-kubernetes/ovnkube-node-8hd6q","openshift-cluster-node-tuning-operator/tuned-xzjmm","openshift-dns/dns-default-qbfrp","openshift-console/console-d6f5dbdb9-6wwq7","openshift-multus/network-metrics-daemon-v5ftn","openshift-image-registry/node-ca-44szm","openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq","kube-system/konnectivity-agent-zsmmj","openshift-monitoring/alertmanager-main-0","openshift-network-operator/iptables-alerter-wlwnn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal","openshift-dns/node-resolver-wzqjr","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-nvttn","openshift-console/console-5c6f844c96-s75ms","openshift-monitoring/metrics-server-7d849c4dfc-tbtqx"] Oct 31 22:50:23.114945 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.114907 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 22:50:23.114945 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.114939 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\"" Oct 31 22:50:23.115146 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.114908 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.115299 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.115274 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.115395 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.115377 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\"" Oct 31 22:50:23.115545 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.115528 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 22:50:23.115591 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.115580 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\"" Oct 31 22:50:23.116091 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.116068 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.116091 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.116082 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.116729 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.116708 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 22:50:23.116883 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.116717 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\"" Oct 31 22:50:23.116999 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.116748 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.118482 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.117686 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119275 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119530 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119636 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119752 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7g25v\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119854 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.119869 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120092 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120140 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120208 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120365 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120395 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120514 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120727 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120779 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120818 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 22:50:23.121094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.120949 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\"" Oct 31 22:50:23.121717 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121693 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 22:50:23.121717 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121696 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 22:50:23.121872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121742 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.121872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121777 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 22:50:23.121872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121813 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 22:50:23.121872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121818 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.121872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121851 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121882 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121889 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121920 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.121967 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122021 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9xrd\"" Oct 31 22:50:23.122114 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122098 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.122411 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122132 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\"" Oct 31 22:50:23.122411 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122210 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122411 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122294 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 22:50:23.122411 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122367 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 22:50:23.122554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122452 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.122633 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122595 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\"" Oct 31 22:50:23.122725 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122641 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.122725 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122663 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 22:50:23.122828 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122724 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\"" Oct 31 22:50:23.122828 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122772 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 22:50:23.122894 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.122841 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\"" Oct 31 22:50:23.124252 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.124231 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.126084 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.126067 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s2g2v\"" Oct 31 22:50:23.126859 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.126838 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 22:50:23.126947 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.126931 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 22:50:23.128128 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.128108 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.201006 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.200967 7074 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 22:50:23.206348 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.206325 7074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:23.206476 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.206409 7074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:23.216965 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.216934 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 22:50:23.217127 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.216982 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 22:50:23.217127 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217026 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 22:50:23.217127 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217076 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 22:50:23.217127 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217090 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217158 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217163 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217218 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217278 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217284 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 22:50:23.217321 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217309 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217323 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217331 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.216940 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217393 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217418 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cbxkt\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217519 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217156 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217519 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217539 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 22:50:23.217585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217586 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217624 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217626 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217659 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217689 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217764 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217898 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 22:50:23.218062 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.217949 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 22:50:23.218397 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.218377 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 22:50:23.218520 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.218468 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qlpwp\"" Oct 31 22:50:23.219032 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219011 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 22:50:23.219123 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219032 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 22:50:23.219919 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219899 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdtc\" (UniqueName: \"kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.220016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219928 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.220016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219949 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.220016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.219974 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.220179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220088 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.220179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220111 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220127 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.220179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220140 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220187 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220148 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220241 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220272 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220304 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220334 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220350 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220350 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.220370 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220364 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220393 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220419 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220450 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220484 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220513 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220522 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n6lmz\"" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220539 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220571 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpck\" (UniqueName: \"kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220596 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220622 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220627 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220649 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220694 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220722 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220751 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzz45\" (UniqueName: \"kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.220786 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220783 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220815 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220844 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220851 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220871 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-serviceca\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220885 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2whgz\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220873 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220913 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h45ml\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.220956 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221056 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221110 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221156 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6c492\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221062 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p9kg6\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221231 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221261 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221302 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221339 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221358 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221392 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221422 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8ps\" (UniqueName: \"kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221442 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-client-certs\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221483 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.221554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221559 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221609 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221652 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgqd\" (UniqueName: \"kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221687 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221703 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221713 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221730 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221848 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221895 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54qb\" (UniqueName: \"kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221915 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221947 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221967 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-cni-binary-copy\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.221971 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222105 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222128 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222144 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222147 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222199 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222220 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222259 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222276 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222295 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222319 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222367 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56499b86-c2fe-429c-88b8-cc5be0e63840-original-pull-secret\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222369 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222414 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222432 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222450 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222472 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-env-overrides\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222475 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.222584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222627 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-tls\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222630 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7phlcd127qi9k\"" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222711 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b307j9jpi8iol\"" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222778 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-script-lib\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222800 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222802 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222875 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222921 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222911 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.222966 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223019 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223048 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223068 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39248804-861a-4442-a717-66a2467e2c72-iptables-alerter-script\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223080 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223342 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9004f5b-8f45-4dfe-b883-d485d3b43582-metrics-tls\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223537 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223734 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.223910 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224018 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224055 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224084 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224116 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.224161 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224149 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224178 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224242 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9004f5b-8f45-4dfe-b883-d485d3b43582-tmp-dir\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224360 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aej5qsglgbkq\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224457 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224470 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-metrics-server-audit-profiles\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224670 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224729 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224792 7074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:23.224829 7074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.224889 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca1b181-263f-43a9-96fb-2a2efbd8c963-metrics-certs\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225090 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225108 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9d5qn\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225143 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 22:50:23.225289 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225289 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225304 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225328 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225358 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225378 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kvll7\"" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225387 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225427 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225456 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225469 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225489 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzddt\" (UniqueName: \"kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225517 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225543 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225608 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225612 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225702 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jwb\" (UniqueName: \"kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225733 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225759 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225783 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225807 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225831 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225856 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225889 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.225924 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226031 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226064 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226090 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226117 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226143 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226152 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb149490-9112-463a-b5ef-107f84f04d0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226167 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226193 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226221 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226247 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47md\" (UniqueName: \"kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226270 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226290 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226322 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226340 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226356 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226374 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226527 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226555 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226580 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226598 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226620 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226644 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226661 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226678 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226705 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226726 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226743 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwqt\" (UniqueName: \"kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226761 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbn7\" (UniqueName: \"kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226789 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226812 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226830 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226847 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226868 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226901 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226921 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226941 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.226969 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227002 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227025 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.227016 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227046 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227064 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227090 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227107 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227125 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvb2\" (UniqueName: \"kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227142 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227160 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227177 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227347 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcqm\" (UniqueName: \"kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227368 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227371 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227386 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227404 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227413 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227421 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227440 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227457 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227469 7074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227476 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227494 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227521 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:23.227525 7074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-157.us-east-2.compute.internal" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227539 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227559 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227575 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227595 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227613 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227631 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227649 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227666 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227683 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227702 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227720 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227736 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227775 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227793 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227811 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227831 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227851 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227871 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227889 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227907 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227926 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227945 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfxx2\" (UniqueName: \"kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227964 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxxz\" (UniqueName: \"kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.227982 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228027 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228046 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228065 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228082 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228100 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228119 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rgk\" (UniqueName: \"kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228138 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228156 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228173 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228193 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvh7r\" (UniqueName: \"kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228215 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4sl8\" (UniqueName: \"kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228511 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-textfile\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228682 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228838 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-daemon-config\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.228936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.228975 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229148 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229389 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229406 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229485 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229548 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229604 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229675 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-cert\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229801 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.229923 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230031 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230078 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovnkube-config\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230220 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98b6f6c7-3878-491d-8a05-131c56390f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230283 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230343 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-client-ca-bundle\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230402 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230437 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230482 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-tuned\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230487 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59b2c63f-8550-4152-9759-d522b6b4e520-konnectivity-ca\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230537 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59b2c63f-8550-4152-9759-d522b6b4e520-agent-certs\") pod \"konnectivity-agent-zsmmj\" (UID: \"59b2c63f-8550-4152-9759-d522b6b4e520\") " pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230552 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230721 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-audit-log\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230732 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230768 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9fe7d02-92f2-41fd-9122-abdfdc159006-ovn-node-metrics-cert\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230774 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.230792 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f91b6ba2-b532-47ef-b1de-ea7a018f2355-tmp\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231171 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231282 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231393 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231415 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2252e42-a7dd-4822-83ca-65aff7bb802a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231475 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb149490-9112-463a-b5ef-107f84f04d0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231591 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231605 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231592 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231698 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231701 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-grpc-tls\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.231963 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231757 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.234180 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231784 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.234180 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231823 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-tmp-dir\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231939 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9004f5b-8f45-4dfe-b883-d485d3b43582-config-volume\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.231976 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb149490-9112-463a-b5ef-107f84f04d0c-data-volume\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232099 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.234257 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpck\" (UniqueName: \"kubernetes.io/projected/26fd3b3c-9c2e-43b7-9660-ec1ff811348e-kube-api-access-grpck\") pod \"ingress-canary-t48z9\" (UID: \"26fd3b3c-9c2e-43b7-9660-ec1ff811348e\") " pod="openshift-ingress-canary/ingress-canary-t48z9" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232127 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-metrics-client-ca\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232274 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232341 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232357 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232491 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232495 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232528 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232535 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.234345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232557 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-tmp\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232639 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-metrics-client-ca\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232705 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232802 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232826 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233003 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233024 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233170 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233228 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233231 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-secret-metrics-server-tls\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233346 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233406 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233456 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdtc\" (UniqueName: \"kubernetes.io/projected/f6b13f57-54cc-4499-a9a5-44eec96c4488-kube-api-access-lrdtc\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.233919 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzz45\" (UniqueName: \"kubernetes.io/projected/c9fe7d02-92f2-41fd-9122-abdfdc159006-kube-api-access-lzz45\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.234057 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8ps\" (UniqueName: \"kubernetes.io/projected/94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f-kube-api-access-bp8ps\") pod \"thanos-querier-5896fc6d85-lpp69\" (UID: \"94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f\") " pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.234057 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54qb\" (UniqueName: \"kubernetes.io/projected/eb149490-9112-463a-b5ef-107f84f04d0c-kube-api-access-z54qb\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.234267 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.234957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.232332 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"console-5c6f844c96-s75ms\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:23.235486 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.235344 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.237968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.237946 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgqd\" (UniqueName: \"kubernetes.io/projected/98b6f6c7-3878-491d-8a05-131c56390f37-kube-api-access-gmgqd\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.238229 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.238211 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzxd\" (UniqueName: \"kubernetes.io/projected/e383e48a-7ee2-4c25-a5e4-d708af31e31b-kube-api-access-dzzxd\") pod \"network-check-target-nvttn\" (UID: \"e383e48a-7ee2-4c25-a5e4-d708af31e31b\") " pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.243584 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jwb\" (UniqueName: \"kubernetes.io/projected/d9004f5b-8f45-4dfe-b883-d485d3b43582-kube-api-access-x6jwb\") pod \"dns-default-qbfrp\" (UID: \"d9004f5b-8f45-4dfe-b883-d485d3b43582\") " pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.243909 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvh7r\" (UniqueName: \"kubernetes.io/projected/2c47d3f0-fedf-456a-b4db-dc5115e9c5d6-kube-api-access-jvh7r\") pod \"openshift-state-metrics-7bd7489957-rvwxq\" (UID: \"2c47d3f0-fedf-456a-b4db-dc5115e9c5d6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rvwxq" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.243934 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcqm\" (UniqueName: \"kubernetes.io/projected/b2252e42-a7dd-4822-83ca-65aff7bb802a-kube-api-access-5fcqm\") pod \"kube-state-metrics-5dc8d7d89b-mzdtc\" (UID: \"b2252e42-a7dd-4822-83ca-65aff7bb802a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mzdtc" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244015 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwqt\" (UniqueName: \"kubernetes.io/projected/39248804-861a-4442-a717-66a2467e2c72-kube-api-access-qhwqt\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244030 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbn7\" (UniqueName: \"kubernetes.io/projected/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-kube-api-access-8kbn7\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244118 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxxz\" (UniqueName: \"kubernetes.io/projected/27c3dce2-82b9-4868-8ec9-2d480ff15a1c-kube-api-access-qwxxz\") pod \"monitoring-plugin-77f4f76594-gfw6p\" (UID: \"27c3dce2-82b9-4868-8ec9-2d480ff15a1c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244397 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47md\" (UniqueName: \"kubernetes.io/projected/7ca1b181-263f-43a9-96fb-2a2efbd8c963-kube-api-access-n47md\") pod \"network-metrics-daemon-v5ftn\" (UID: \"7ca1b181-263f-43a9-96fb-2a2efbd8c963\") " pod="openshift-multus/network-metrics-daemon-v5ftn" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244525 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rgk\" (UniqueName: \"kubernetes.io/projected/f91b6ba2-b532-47ef-b1de-ea7a018f2355-kube-api-access-j9rgk\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244633 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfxx2\" (UniqueName: \"kubernetes.io/projected/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kube-api-access-gfxx2\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244729 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvb2\" (UniqueName: \"kubernetes.io/projected/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-kube-api-access-dkvb2\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244846 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"console-d6f5dbdb9-6wwq7\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:23.245020 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.244869 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"prometheus-k8s-0\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:23.245648 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.245106 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"alertmanager-main-0\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:23.246558 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.246537 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4sl8\" (UniqueName: \"kubernetes.io/projected/cc9ed1c1-75ac-42cc-8613-67993c17b9a5-kube-api-access-f4sl8\") pod \"metrics-server-7d849c4dfc-tbtqx\" (UID: \"cc9ed1c1-75ac-42cc-8613-67993c17b9a5\") " pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:23.248859 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.248840 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzddt\" (UniqueName: \"kubernetes.io/projected/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-kube-api-access-mzddt\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.328711 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328667 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328755 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-var-lib-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328790 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328815 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328829 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328845 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.328884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328883 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-k8s-cni-cncf-io\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328902 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328912 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-systemd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.328962 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329016 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329031 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-modprobe-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329049 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb149490-9112-463a-b5ef-107f84f04d0c-crio-socket\") pod \"insights-runtime-extractor-j2gk2\" (UID: \"eb149490-9112-463a-b5ef-107f84f04d0c\") " pod="openshift-insights/insights-runtime-extractor-j2gk2" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329086 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-cnibin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329088 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329124 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329143 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329166 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329200 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329198 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329208 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329219 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-conf-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329224 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329249 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329257 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-host\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329224 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-multus-certs\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329273 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-dbus\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329277 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329225 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-run\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329307 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329329 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329345 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329346 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-lib-modules\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329341 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-netd\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329362 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329382 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-log-socket\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329386 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329394 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-hostroot\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329392 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-cni-bin\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329391 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329439 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-registration-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329440 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329475 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329477 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-var-lib-kubelet\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329500 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329523 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-node-log\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329536 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329545 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329554 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329571 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329587 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329604 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329618 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329630 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-system-cni-dir\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329643 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-cnibin\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329649 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329659 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-socket-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329671 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56499b86-c2fe-429c-88b8-cc5be0e63840-kubelet-config\") pod \"global-pull-secret-syncer-qqjd2\" (UID: \"56499b86-c2fe-429c-88b8-cc5be0e63840\") " pod="kube-system/global-pull-secret-syncer-qqjd2" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329670 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-run-netns\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329683 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98b6f6c7-3878-491d-8a05-131c56390f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ctfvq\" (UID: \"98b6f6c7-3878-491d-8a05-131c56390f37\") " pod="openshift-multus/multus-additional-cni-plugins-ctfvq" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329683 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329702 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329712 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.329706 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329729 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329751 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329774 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-multus-socket-dir-parent\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329777 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329782 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-conf\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329777 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-root\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329804 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329811 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-node-exporter-wtmp\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329816 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysconfig\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329836 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329847 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-sys\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329855 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329870 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329885 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329899 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-etc-openvswitch\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329886 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-kubelet\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329907 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329903 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-bin\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329932 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329930 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-kubelet\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329950 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-etc-selinux\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329977 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.329980 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-sys-fs\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330017 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330035 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330042 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-run-ovn\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330049 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330064 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-run-netns\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330069 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d-host\") pod \"node-ca-44szm\" (UID: \"4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d\") " pod="openshift-image-registry/node-ca-44szm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330080 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330093 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-systemd\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330105 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330115 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-kubernetes\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330133 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330151 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330166 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330177 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-etc-kubernetes\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330184 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330190 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-host-slash\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330201 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b13f57-54cc-4499-a9a5-44eec96c4488-host-var-lib-cni-multus\") pod \"multus-rdsnf\" (UID: \"f6b13f57-54cc-4499-a9a5-44eec96c4488\") " pod="openshift-multus/multus-rdsnf" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330208 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330211 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39248804-861a-4442-a717-66a2467e2c72-host-slash\") pod \"iptables-alerter-wlwnn\" (UID: \"39248804-861a-4442-a717-66a2467e2c72\") " pod="openshift-network-operator/iptables-alerter-wlwnn" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330230 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330245 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330269 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9fe7d02-92f2-41fd-9122-abdfdc159006-systemd-units\") pod \"ovnkube-node-8hd6q\" (UID: \"c9fe7d02-92f2-41fd-9122-abdfdc159006\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330294 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d187ad0-d0cd-4442-a0b5-ab7e324cacda-sys\") pod \"node-exporter-7wpwb\" (UID: \"0d187ad0-d0cd-4442-a0b5-ab7e324cacda\") " pod="openshift-monitoring/node-exporter-7wpwb" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330298 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-device-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330289 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f91b6ba2-b532-47ef-b1de-ea7a018f2355-etc-sysctl-d\") pod \"tuned-xzjmm\" (UID: \"f91b6ba2-b532-47ef-b1de-ea7a018f2355\") " pod="openshift-cluster-node-tuning-operator/tuned-xzjmm" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330327 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330346 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330399 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48ac9bdf-c637-46e4-9fac-53c43f3fd161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m56nk\" (UID: \"48ac9bdf-c637-46e4-9fac-53c43f3fd161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m56nk" Oct 31 22:50:23.331008 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:23.330397 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25ef2f42-14fa-4d62-be14-ef9fe9fd92f5-hosts-file\") pod \"node-resolver-wzqjr\" (UID: \"25ef2f42-14fa-4d62-be14-ef9fe9fd92f5\") " pod="openshift-dns/node-resolver-wzqjr" Oct 31 22:50:24.034397 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:24.034367 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5896fc6d85-lpp69" Oct 31 22:50:24.262713 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:24.262686 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qbfrp" Oct 31 22:50:25.045793 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:25.045761 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:28.033754 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:28.033724 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:29.149149 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:29.149122 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:29.149618 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:29.149313 7074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:50:29.749924 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:29.749892 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gfw6p" Oct 31 22:50:30.352113 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:30.352073 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:31.144055 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:31.144026 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hd6q" Oct 31 22:50:32.800054 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:32.800025 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:50:39.217519 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:39.217482 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d849c4dfc-tbtqx" Oct 31 22:50:44.575181 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575144 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575419 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="alertmanager" containerID="cri-o://1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71" gracePeriod=120 Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575485 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-metric" containerID="cri-o://10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a" gracePeriod=120 Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575538 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-web" containerID="cri-o://628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d" gracePeriod=120 Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575560 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="prom-label-proxy" containerID="cri-o://a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93" gracePeriod=120 Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575617 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="config-reloader" containerID="cri-o://1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1" gracePeriod=120 Oct 31 22:50:44.575724 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:44.575567 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy" containerID="cri-o://88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa" gracePeriod=120 Oct 31 22:50:44.887233 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:44.887153 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:50:45.296504 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296475 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93" exitCode=0 Oct 31 22:50:45.296504 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296499 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a" exitCode=0 Oct 31 22:50:45.296504 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296507 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa" exitCode=0 Oct 31 22:50:45.296504 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296513 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1" exitCode=0 Oct 31 22:50:45.296504 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296519 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71" exitCode=0 Oct 31 22:50:45.296800 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296557 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93"} Oct 31 22:50:45.296800 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296624 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a"} Oct 31 22:50:45.296800 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296651 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa"} Oct 31 22:50:45.296800 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296676 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1"} Oct 31 22:50:45.296800 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.296700 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71"} Oct 31 22:50:45.817332 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.817307 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:45.835326 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835291 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835465 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835336 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835465 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835363 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835465 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835389 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835465 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835411 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835465 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835435 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835731 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835472 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835731 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835511 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835731 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835545 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835731 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835574 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.835731 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.835631 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.836587 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.836190 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:45.836587 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.836540 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:45.839241 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839199 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.839351 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839230 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "kube-api-access-cpmpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:45.839419 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839371 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:45.839476 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839449 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:45.839573 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839549 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.839685 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839669 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.839737 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.839725 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.843859 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.843838 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.850555 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.850527 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:45.936900 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.936824 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.936900 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.936865 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\" (UID: \"dc61c3ce-5106-4bbd-b69d-e3aa762aba20\") " Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937010 7074 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937021 7074 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937031 7074 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937040 7074 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-tls-assets\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937070 7074 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-main-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937080 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmpb\" (UniqueName: \"kubernetes.io/projected/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-kube-api-access-cpmpb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937090 7074 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-config-out\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937098 7074 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-cluster-tls-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937107 7074 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-web-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937115 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-metrics-client-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937123 7074 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-main-db\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:45.937147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.937135 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:45.938852 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:45.938830 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric" pod "dc61c3ce-5106-4bbd-b69d-e3aa762aba20" (UID: "dc61c3ce-5106-4bbd-b69d-e3aa762aba20"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:46.038339 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.038301 7074 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:46.038339 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.038334 7074 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc61c3ce-5106-4bbd-b69d-e3aa762aba20-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:46.303774 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.303739 7074 generic.go:356] "Generic (PLEG): container finished" podID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerID="628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d" exitCode=0 Oct 31 22:50:46.303954 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.303824 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.303954 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.303823 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d"} Oct 31 22:50:46.303954 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.303899 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc61c3ce-5106-4bbd-b69d-e3aa762aba20","Type":"ContainerDied","Data":"b3a12457099091d18b076ebd85a363e4ce40d499d3cdbfdb454cb53437307468"} Oct 31 22:50:46.303954 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.303931 7074 scope.go:122] "RemoveContainer" containerID="a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93" Oct 31 22:50:46.311594 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.311568 7074 scope.go:122] "RemoveContainer" containerID="10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a" Oct 31 22:50:46.318673 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.318653 7074 scope.go:122] "RemoveContainer" containerID="88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa" Oct 31 22:50:46.326354 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.326333 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:46.326443 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.326430 7074 scope.go:122] "RemoveContainer" containerID="628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d" Oct 31 22:50:46.330818 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.330797 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:46.334273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.334252 7074 scope.go:122] "RemoveContainer" containerID="1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1" Oct 31 22:50:46.341153 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.341137 7074 scope.go:122] "RemoveContainer" containerID="1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71" Oct 31 22:50:46.348170 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.348152 7074 scope.go:122] "RemoveContainer" containerID="b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c" Oct 31 22:50:46.355353 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.355336 7074 scope.go:122] "RemoveContainer" containerID="a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93" Oct 31 22:50:46.356280 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.356247 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93\": container with ID starting with a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93 not found: ID does not exist" containerID="a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93" Oct 31 22:50:46.356372 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356291 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93"} err="failed to get container status \"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93\": rpc error: code = NotFound desc = could not find container \"a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93\": container with ID starting with a9fd06a9711e474de97d7dcc36f758445c7426b12edec487844fc6919018cd93 not found: ID does not exist" Oct 31 22:50:46.356372 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356335 7074 scope.go:122] "RemoveContainer" containerID="10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a" Oct 31 22:50:46.356668 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.356647 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a\": container with ID starting with 10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a not found: ID does not exist" containerID="10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a" Oct 31 22:50:46.356732 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356674 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a"} err="failed to get container status \"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a\": rpc error: code = NotFound desc = could not find container \"10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a\": container with ID starting with 10500aa019c429d5592ed6872079c7294b88bf67e5cac37d5d07a8c37d76a29a not found: ID does not exist" Oct 31 22:50:46.356732 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356690 7074 scope.go:122] "RemoveContainer" containerID="88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa" Oct 31 22:50:46.356947 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.356928 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa\": container with ID starting with 88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa not found: ID does not exist" containerID="88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa" Oct 31 22:50:46.357025 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356952 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa"} err="failed to get container status \"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa\": rpc error: code = NotFound desc = could not find container \"88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa\": container with ID starting with 88f82e326ece0ad4cb31d8bec98f166b5ba0023b0fb94c4d9f420c1167e0e5fa not found: ID does not exist" Oct 31 22:50:46.357025 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.356964 7074 scope.go:122] "RemoveContainer" containerID="628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d" Oct 31 22:50:46.357301 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.357275 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d\": container with ID starting with 628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d not found: ID does not exist" containerID="628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d" Oct 31 22:50:46.357354 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357310 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d"} err="failed to get container status \"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d\": rpc error: code = NotFound desc = could not find container \"628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d\": container with ID starting with 628fa9986965811b63a7f5d82a1c9aab110e6c3a5c428542cf657ecb5dc0928d not found: ID does not exist" Oct 31 22:50:46.357354 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357332 7074 scope.go:122] "RemoveContainer" containerID="1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1" Oct 31 22:50:46.357631 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.357612 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1\": container with ID starting with 1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1 not found: ID does not exist" containerID="1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1" Oct 31 22:50:46.357710 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357639 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1"} err="failed to get container status \"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1\": rpc error: code = NotFound desc = could not find container \"1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1\": container with ID starting with 1531b2557df338b57b80dd3f054b0e4819e65761d4c57549ec6ff1bd4f7943a1 not found: ID does not exist" Oct 31 22:50:46.357710 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357660 7074 scope.go:122] "RemoveContainer" containerID="1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71" Oct 31 22:50:46.357925 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.357908 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71\": container with ID starting with 1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71 not found: ID does not exist" containerID="1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71" Oct 31 22:50:46.358024 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357931 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71"} err="failed to get container status \"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71\": rpc error: code = NotFound desc = could not find container \"1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71\": container with ID starting with 1e772563ee97722b568360d0518f88e2da4b8c93decbc568e550a70ab8999d71 not found: ID does not exist" Oct 31 22:50:46.358024 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.357950 7074 scope.go:122] "RemoveContainer" containerID="b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c" Oct 31 22:50:46.358238 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:46.358222 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c\": container with ID starting with b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c not found: ID does not exist" containerID="b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c" Oct 31 22:50:46.358292 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.358243 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c"} err="failed to get container status \"b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c\": rpc error: code = NotFound desc = could not find container \"b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c\": container with ID starting with b27d9de599ddb19805441114bd49eeb8f5836caeb9ae36a2084a844e2df5183c not found: ID does not exist" Oct 31 22:50:46.363258 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363238 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:46.363535 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363516 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="alertmanager" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363536 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="alertmanager" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363556 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-web" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363561 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-web" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363569 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="config-reloader" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363574 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="config-reloader" Oct 31 22:50:46.363585 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363584 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-metric" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363589 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-metric" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363598 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="init-config-reloader" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363603 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="init-config-reloader" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363613 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="prom-label-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363618 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="prom-label-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363624 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363629 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363675 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="config-reloader" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363688 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-metric" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363696 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="init-config-reloader" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363702 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="prom-label-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363708 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363714 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="alertmanager" Oct 31 22:50:46.363782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.363721 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" containerName="kube-rbac-proxy-web" Oct 31 22:50:46.365972 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.365955 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.370576 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370525 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 22:50:46.370683 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370537 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 22:50:46.370683 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370546 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 22:50:46.370683 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370549 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 22:50:46.370683 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370603 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 22:50:46.370896 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370713 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 22:50:46.370896 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370822 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 22:50:46.370896 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370821 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 22:50:46.370896 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.370863 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h45ml\"" Oct 31 22:50:46.372854 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.372785 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 22:50:46.385334 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.385307 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:46.443484 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443453 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443484 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443489 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443689 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443509 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443689 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443613 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443689 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443631 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-web-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443714 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443751 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443773 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443898 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443856 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s794s\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-kube-api-access-s794s\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443898 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443881 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443972 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443937 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-config-out\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.443972 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443955 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.444075 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.443972 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544569 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544529 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544741 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544579 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544741 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544692 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544840 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544744 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s794s\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-kube-api-access-s794s\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544840 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544763 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.544840 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544801 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-config-out\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545100 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545080 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545178 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545116 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545178 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.544980 7074 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 22:50:46.545178 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545148 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545178 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545177 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545384 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545210 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545384 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545260 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545384 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545289 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-web-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.545527 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.545513 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.546133 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.546049 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.546413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.546197 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd5ca891-558a-46f3-9693-b4a28951d091-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548154 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548130 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548422 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548379 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548422 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548404 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548422 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548412 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548645 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548577 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548645 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548626 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-web-config\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548753 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548669 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.548807 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.548786 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd5ca891-558a-46f3-9693-b4a28951d091-config-out\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.549409 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.549390 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd5ca891-558a-46f3-9693-b4a28951d091-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.558789 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.558735 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s794s\" (UniqueName: \"kubernetes.io/projected/dd5ca891-558a-46f3-9693-b4a28951d091-kube-api-access-s794s\") pod \"alertmanager-main-0\" (UID: \"dd5ca891-558a-46f3-9693-b4a28951d091\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.682365 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.682329 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 22:50:46.974883 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:46.974806 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 22:50:46.977938 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:46.977904 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5ca891_558a_46f3_9693_b4a28951d091.slice/crio-875a4af4b9b26813f475f233a0dbc728743cfdd9ea224478522581892c8b0e49 WatchSource:0}: Error finding container 875a4af4b9b26813f475f233a0dbc728743cfdd9ea224478522581892c8b0e49: Status 404 returned error can't find the container with id 875a4af4b9b26813f475f233a0dbc728743cfdd9ea224478522581892c8b0e49 Oct 31 22:50:47.309336 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:47.309300 7074 generic.go:356] "Generic (PLEG): container finished" podID="dd5ca891-558a-46f3-9693-b4a28951d091" containerID="311ee4796817e8c7b4133acf78c39091aee3ab1fba232162db64467318c5d962" exitCode=0 Oct 31 22:50:47.309534 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:47.309385 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerDied","Data":"311ee4796817e8c7b4133acf78c39091aee3ab1fba232162db64467318c5d962"} Oct 31 22:50:47.309534 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:47.309461 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"875a4af4b9b26813f475f233a0dbc728743cfdd9ea224478522581892c8b0e49"} Oct 31 22:50:48.139517 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.139480 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc61c3ce-5106-4bbd-b69d-e3aa762aba20" path="/var/lib/kubelet/pods/dc61c3ce-5106-4bbd-b69d-e3aa762aba20/volumes" Oct 31 22:50:48.142529 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.142500 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c6f844c96-s75ms" podUID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" containerName="console" containerID="cri-o://a946ee60240b8a747f2611cb0f734552efa513073ed6f100aa9224df8f3c8205" gracePeriod=15 Oct 31 22:50:48.319921 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.319891 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"be0f75bbeb1f0e4d365fed1b004e12a717e1d35a030ebc280c3dd1b2bfcb9991"} Oct 31 22:50:48.320109 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.319971 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"5a8af97105173e4909430888d58b95d0a3b851bc9e7181e38d117a8d9ce29a54"} Oct 31 22:50:48.320109 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.320028 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"d6b6e7c557724ac0ed96cbc44d34851c0eae9dabe9b7f7bec2e519fe24ddff2f"} Oct 31 22:50:48.320109 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.320071 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"9a16140a65d502dec91d78517fe2056f0a82ccded9cdd0743c59507d0cfdea5e"} Oct 31 22:50:48.320109 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.320105 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"2754042a18af75e2df8778f05b783bb8b9ffb1d14fa69ef66ab03c9f6a73490e"} Oct 31 22:50:48.320283 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.320143 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd5ca891-558a-46f3-9693-b4a28951d091","Type":"ContainerStarted","Data":"067cc64677e839785ccb738ef1202eac72f61e743e7a6e4ecd890b74d3ad65cc"} Oct 31 22:50:48.321709 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.321683 7074 generic.go:356] "Generic (PLEG): container finished" podID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" containerID="a946ee60240b8a747f2611cb0f734552efa513073ed6f100aa9224df8f3c8205" exitCode=0 Oct 31 22:50:48.321805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.321740 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6f844c96-s75ms" event={"ID":"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e","Type":"ContainerDied","Data":"a946ee60240b8a747f2611cb0f734552efa513073ed6f100aa9224df8f3c8205"} Oct 31 22:50:48.357135 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.357071 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.346315869 podStartE2EDuration="2.357056925s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.741056ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:48.356389759 +0000 UTC m=+26.339132134" watchObservedRunningTime="2026-10-31 22:50:48.357056925 +0000 UTC m=+26.339799297" Oct 31 22:50:48.400256 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.400188 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:48.464126 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464095 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464284 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464164 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464284 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464189 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464284 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464237 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464284 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464259 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464301 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464326 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464367 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") pod \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\" (UID: \"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e\") " Oct 31 22:50:48.464494 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464436 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:48.464675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464600 7074 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-tmp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.464762 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464638 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:48.464812 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464763 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:48.464812 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464738 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:48.464897 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.464820 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:48.466819 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.466791 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "kube-api-access-b9cf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:48.466819 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.466798 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:48.466939 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.466850 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config" pod "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" (UID: "ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:48.565540 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565507 7074 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565540 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565534 7074 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-oauth-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565540 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565545 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cf2\" (UniqueName: \"kubernetes.io/projected/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-kube-api-access-b9cf2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565746 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565554 7074 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565746 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565563 7074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565746 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565571 7074 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-console-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.565746 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.565580 7074 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e-oauth-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:48.582650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.582615 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dc4c664b9-xj859"] Oct 31 22:50:48.582939 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.582927 7074 cpu_manager.go:414] "containerMap: removing container" podUID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" containerName="console" Oct 31 22:50:48.582977 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.582941 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" containerName="console" Oct 31 22:50:48.583043 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.583034 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" containerName="console" Oct 31 22:50:48.585616 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.585599 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.589619 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589591 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 22:50:48.589619 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589606 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 22:50:48.589789 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589654 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 22:50:48.589845 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589800 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 22:50:48.589845 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589813 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g2hz9\"" Oct 31 22:50:48.589930 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.589920 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 22:50:48.593315 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.593296 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 22:50:48.601811 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.601789 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dc4c664b9-xj859"] Oct 31 22:50:48.666345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666265 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666298 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666345 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666321 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-serving-certs-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666377 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpd68\" (UniqueName: \"kubernetes.io/projected/bc184cbe-a1dd-439c-a678-c3fe086ef90d-kube-api-access-fpd68\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666494 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-metrics-client-ca\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666534 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-federate-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666626 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666569 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.666659 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.666634 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767157 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767125 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767172 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767193 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767217 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-serving-certs-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767234 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpd68\" (UniqueName: \"kubernetes.io/projected/bc184cbe-a1dd-439c-a678-c3fe086ef90d-kube-api-access-fpd68\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767261 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-metrics-client-ca\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767302 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767296 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-federate-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.767614 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.767341 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.768248 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.768215 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-metrics-client-ca\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.768398 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.768379 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.768618 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.768592 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc184cbe-a1dd-439c-a678-c3fe086ef90d-serving-certs-ca-bundle\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.769864 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.769833 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.769981 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.769965 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-federate-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.770232 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.770215 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-telemeter-client-tls\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.770292 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.770270 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc184cbe-a1dd-439c-a678-c3fe086ef90d-secret-telemeter-client\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.779605 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.779586 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpd68\" (UniqueName: \"kubernetes.io/projected/bc184cbe-a1dd-439c-a678-c3fe086ef90d-kube-api-access-fpd68\") pod \"telemeter-client-5dc4c664b9-xj859\" (UID: \"bc184cbe-a1dd-439c-a678-c3fe086ef90d\") " pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.898273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.898243 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" Oct 31 22:50:48.980571 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980539 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:48.980838 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980805 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="prometheus" containerID="cri-o://90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" gracePeriod=600 Oct 31 22:50:48.980968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980827 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy" containerID="cri-o://14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb" gracePeriod=600 Oct 31 22:50:48.980968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980845 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-web" containerID="cri-o://5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4" gracePeriod=600 Oct 31 22:50:48.980968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980843 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="thanos-sidecar" containerID="cri-o://0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d" gracePeriod=600 Oct 31 22:50:48.980968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980875 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="config-reloader" containerID="cri-o://de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9" gracePeriod=600 Oct 31 22:50:48.980968 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:48.980883 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725" gracePeriod=600 Oct 31 22:50:49.178299 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.178046 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dc4c664b9-xj859"] Oct 31 22:50:49.180574 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:49.180551 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc184cbe_a1dd_439c_a678_c3fe086ef90d.slice/crio-c130583c1d769c31718ff9f51e3c40ae3d2b965e1a7b2bc191254e113ee278b0 WatchSource:0}: Error finding container c130583c1d769c31718ff9f51e3c40ae3d2b965e1a7b2bc191254e113ee278b0: Status 404 returned error can't find the container with id c130583c1d769c31718ff9f51e3c40ae3d2b965e1a7b2bc191254e113ee278b0 Oct 31 22:50:49.182573 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.182556 7074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:50:49.326526 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.326484 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" event={"ID":"bc184cbe-a1dd-439c-a678-c3fe086ef90d","Type":"ContainerStarted","Data":"c130583c1d769c31718ff9f51e3c40ae3d2b965e1a7b2bc191254e113ee278b0"} Oct 31 22:50:49.329717 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329694 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725" exitCode=0 Oct 31 22:50:49.329717 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329714 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb" exitCode=0 Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329722 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d" exitCode=0 Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329728 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9" exitCode=0 Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329736 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" exitCode=0 Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329766 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725"} Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329831 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb"} Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329871 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d"} Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329895 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9"} Oct 31 22:50:49.329957 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.329917 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1"} Oct 31 22:50:49.331136 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.331119 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6f844c96-s75ms" Oct 31 22:50:49.331240 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.331133 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6f844c96-s75ms" event={"ID":"ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e","Type":"ContainerDied","Data":"6969d27a33a7e4170db65829435bbe8a91c5f035b6d36a0f81e0f0baef60a397"} Oct 31 22:50:49.331240 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.331169 7074 scope.go:122] "RemoveContainer" containerID="a946ee60240b8a747f2611cb0f734552efa513073ed6f100aa9224df8f3c8205" Oct 31 22:50:49.352467 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.352442 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:50:49.356974 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:49.356955 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c6f844c96-s75ms"] Oct 31 22:50:50.031183 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.031143 7074 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 22:50:50.031458 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.031214 7074 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 22:50:50.031458 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.031241 7074 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 22:50:50.031458 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.031257 7074 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 22:50:50.141698 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.141662 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e" path="/var/lib/kubelet/pods/ccc12ea2-4cb4-4f1f-9b6b-157c5b0c403e/volumes" Oct 31 22:50:50.243252 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.243221 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.281637 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281556 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281637 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281601 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281848 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281660 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281848 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281692 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281848 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281729 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281848 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281769 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.281848 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281799 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282130 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281849 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282130 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281883 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282130 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.281978 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.282130 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282049 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282130 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282092 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282180 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282218 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282282 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282316 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282346 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282374 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282405 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282434 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282433 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282441 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282476 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") pod \"72ec9d28-478d-45f2-82f5-8197104db046\" (UID: \"72ec9d28-478d-45f2-82f5-8197104db046\") " Oct 31 22:50:50.282675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282554 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.282965 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282782 7074 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.282965 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282801 7074 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.282965 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282817 7074 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.282965 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.282832 7074 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.283205 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.283071 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.283205 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.283188 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:50.284061 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.283778 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.284842 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.284813 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:50.286592 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.286553 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.286811 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.286780 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.287135 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.287110 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.289352 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.289315 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:50.289499 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.289466 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.290513 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.290466 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.290687 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.290660 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.291094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.290969 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.291094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.291055 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:50.291290 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.291265 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "kube-api-access-ddkpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:50.291938 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.291912 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.302196 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.302170 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config" pod "72ec9d28-478d-45f2-82f5-8197104db046" (UID: "72ec9d28-478d-45f2-82f5-8197104db046"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:50.339737 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.339703 7074 generic.go:356] "Generic (PLEG): container finished" podID="72ec9d28-478d-45f2-82f5-8197104db046" containerID="5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4" exitCode=0 Oct 31 22:50:50.339909 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.339796 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.339909 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.339794 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4"} Oct 31 22:50:50.339909 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.339882 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72ec9d28-478d-45f2-82f5-8197104db046","Type":"ContainerDied","Data":"1107aefd212ccb47556fa80135a234463ef9487119ff39d8f94c1031cf019ef6"} Oct 31 22:50:50.340090 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.339932 7074 scope.go:122] "RemoveContainer" containerID="9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725" Oct 31 22:50:50.351764 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.351743 7074 scope.go:122] "RemoveContainer" containerID="14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb" Oct 31 22:50:50.354317 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.354299 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nvttn" Oct 31 22:50:50.360458 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.360438 7074 scope.go:122] "RemoveContainer" containerID="5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4" Oct 31 22:50:50.368225 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.368205 7074 scope.go:122] "RemoveContainer" containerID="0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d" Oct 31 22:50:50.372300 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.372280 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:50.376751 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.376730 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:50.376817 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.376780 7074 scope.go:122] "RemoveContainer" containerID="de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9" Oct 31 22:50:50.383288 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383261 7074 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383290 7074 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383303 7074 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-metrics-client-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383319 7074 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-db\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383331 7074 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-web-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383340 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddkpr\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-kube-api-access-ddkpr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383348 7074 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-configmap-metrics-client-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383357 7074 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383368 7074 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-grpc-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383382 7074 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383394 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383394 7074 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383770 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383404 7074 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383770 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383413 7074 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72ec9d28-478d-45f2-82f5-8197104db046-secret-kube-rbac-proxy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383770 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383422 7074 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72ec9d28-478d-45f2-82f5-8197104db046-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383770 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383430 7074 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72ec9d28-478d-45f2-82f5-8197104db046-tls-assets\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.383770 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.383438 7074 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72ec9d28-478d-45f2-82f5-8197104db046-config-out\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:50.386399 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.386370 7074 scope.go:122] "RemoveContainer" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" Oct 31 22:50:50.395293 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.395274 7074 scope.go:122] "RemoveContainer" containerID="fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f" Oct 31 22:50:50.408674 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.408588 7074 scope.go:122] "RemoveContainer" containerID="9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725" Oct 31 22:50:50.409339 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.409312 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725\": container with ID starting with 9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725 not found: ID does not exist" containerID="9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725" Oct 31 22:50:50.409452 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.409350 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725"} err="failed to get container status \"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725\": rpc error: code = NotFound desc = could not find container \"9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725\": container with ID starting with 9c4cc433baa7ee2a00a9ef1d9b3b5d2dca3f94aa35a84e821b325f8dbf63f725 not found: ID does not exist" Oct 31 22:50:50.409452 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.409376 7074 scope.go:122] "RemoveContainer" containerID="14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb" Oct 31 22:50:50.409696 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.409672 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:50.409779 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.409683 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb\": container with ID starting with 14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb not found: ID does not exist" containerID="14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb" Oct 31 22:50:50.409850 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.409791 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb"} err="failed to get container status \"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb\": rpc error: code = NotFound desc = could not find container \"14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb\": container with ID starting with 14b3c50123de739e70ac0b06b3b4d0b7a25738ccc30ed78cb811e09ea7b61ceb not found: ID does not exist" Oct 31 22:50:50.409850 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.409808 7074 scope.go:122] "RemoveContainer" containerID="5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4" Oct 31 22:50:50.410134 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.410108 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4\": container with ID starting with 5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4 not found: ID does not exist" containerID="5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4" Oct 31 22:50:50.410219 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410142 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4"} err="failed to get container status \"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4\": rpc error: code = NotFound desc = could not find container \"5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4\": container with ID starting with 5cd989afa980b8773277df2a77a23ccd9129583d7b52877e150392c4801b3ab4 not found: ID does not exist" Oct 31 22:50:50.410219 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410163 7074 scope.go:122] "RemoveContainer" containerID="0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d" Oct 31 22:50:50.410219 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410207 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="thanos-sidecar" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410225 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="thanos-sidecar" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410249 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="config-reloader" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410258 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="config-reloader" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410270 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="init-config-reloader" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410278 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="init-config-reloader" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410295 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410303 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410316 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-web" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410324 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-web" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410340 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="prometheus" Oct 31 22:50:50.410364 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410347 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="prometheus" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410371 7074 cpu_manager.go:414] "containerMap: removing container" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-thanos" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410379 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-thanos" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410448 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-thanos" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.410435 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d\": container with ID starting with 0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d not found: ID does not exist" containerID="0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410485 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d"} err="failed to get container status \"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d\": rpc error: code = NotFound desc = could not find container \"0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d\": container with ID starting with 0366651ca12a54f1a3f6b56233bcd79fd9e7a399d769bdb38699d27d5300741d not found: ID does not exist" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410505 7074 scope.go:122] "RemoveContainer" containerID="de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410461 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="thanos-sidecar" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410554 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="config-reloader" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410564 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="init-config-reloader" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410575 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410583 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="kube-rbac-proxy-web" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410592 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ec9d28-478d-45f2-82f5-8197104db046" containerName="prometheus" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.410736 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9\": container with ID starting with de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9 not found: ID does not exist" containerID="de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410760 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9"} err="failed to get container status \"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9\": rpc error: code = NotFound desc = could not find container \"de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9\": container with ID starting with de0b3426ccef7ad90fa49b6a15d1c7a1bbd607eba6fbbe9915bc485085bb62e9 not found: ID does not exist" Oct 31 22:50:50.410851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.410780 7074 scope.go:122] "RemoveContainer" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" Oct 31 22:50:50.411453 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.411038 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1\": container with ID starting with 90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1 not found: ID does not exist" containerID="90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1" Oct 31 22:50:50.411453 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.411065 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1"} err="failed to get container status \"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1\": rpc error: code = NotFound desc = could not find container \"90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1\": container with ID starting with 90417b00df9b7332e8fb754ffe6395fd96f5daa857030cf787745ab1ee0812f1 not found: ID does not exist" Oct 31 22:50:50.411453 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.411085 7074 scope.go:122] "RemoveContainer" containerID="fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f" Oct 31 22:50:50.411453 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:50:50.411362 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f\": container with ID starting with fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f not found: ID does not exist" containerID="fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f" Oct 31 22:50:50.411453 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.411394 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f"} err="failed to get container status \"fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f\": rpc error: code = NotFound desc = could not find container \"fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f\": container with ID starting with fd3fd3acea31d0bdd4020b31ecb415f60e108afbe7059757fff7ddba014cd37f not found: ID does not exist" Oct 31 22:50:50.414938 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.414920 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.419066 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.419047 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 22:50:50.419187 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.419175 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 22:50:50.419239 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.419229 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 22:50:50.419281 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.419262 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aej5qsglgbkq\"" Oct 31 22:50:50.420223 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420203 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 22:50:50.420316 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420292 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 22:50:50.420408 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420379 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 22:50:50.420468 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420461 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 22:50:50.420857 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420840 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 22:50:50.421012 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420918 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 22:50:50.421012 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.420969 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9d5qn\"" Oct 31 22:50:50.421151 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.421057 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 22:50:50.422191 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.422172 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 22:50:50.422316 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.422300 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 22:50:50.422382 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.422304 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 22:50:50.432073 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.431876 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:50.433047 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.432704 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 22:50:50.484744 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.484712 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.484888 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.484752 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.484888 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.484773 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485035 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.484915 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485035 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.484974 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485119 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485062 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485119 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485089 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485119 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485108 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485245 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485164 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485245 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485213 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzxt\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-kube-api-access-qkzxt\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485314 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485253 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485314 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485293 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485314 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485310 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485426 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485338 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485426 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485396 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485528 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485445 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485528 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485513 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485621 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485547 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485621 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485578 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.485681 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.485619 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586383 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586301 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586383 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586349 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586383 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586375 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzxt\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-kube-api-access-qkzxt\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586408 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586448 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586467 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586489 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586507 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586525 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586559 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586578 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586600 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586617 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586646 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.586665 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586672 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586690 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586712 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586746 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586769 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.586792 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587573 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.587337 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.587871 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.587845 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.588602 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.588580 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.589669 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.589644 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.589867 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.589849 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.589944 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.589897 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.590859 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.590831 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.591286 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.591260 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.591650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.591625 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8324d2f9-f4bd-451a-925b-b6e9657b46d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.593294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.592972 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.593294 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.593070 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.593675 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.593649 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.594589 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.594566 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.595084 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.595048 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.595329 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.595306 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.595622 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.595593 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.595709 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.595688 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.595975 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.595959 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.596547 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.596527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8324d2f9-f4bd-451a-925b-b6e9657b46d2-config\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.600717 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.600698 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzxt\" (UniqueName: \"kubernetes.io/projected/8324d2f9-f4bd-451a-925b-b6e9657b46d2-kube-api-access-qkzxt\") pod \"prometheus-k8s-0\" (UID: \"8324d2f9-f4bd-451a-925b-b6e9657b46d2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:50.750738 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:50.750708 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:50:51.050039 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.049427 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 22:50:51.056296 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:50:51.056271 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8324d2f9_f4bd_451a_925b_b6e9657b46d2.slice/crio-4823cb70f5c5f9288678441dce23411b86f53cc1b1532ceb4aa6c0ebc7967493 WatchSource:0}: Error finding container 4823cb70f5c5f9288678441dce23411b86f53cc1b1532ceb4aa6c0ebc7967493: Status 404 returned error can't find the container with id 4823cb70f5c5f9288678441dce23411b86f53cc1b1532ceb4aa6c0ebc7967493 Oct 31 22:50:51.348591 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.348552 7074 generic.go:356] "Generic (PLEG): container finished" podID="8324d2f9-f4bd-451a-925b-b6e9657b46d2" containerID="728400baf3da2e3b0ab588ab57cf9de25f3c39c8d964a02160d446779645affd" exitCode=0 Oct 31 22:50:51.349078 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.348638 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerDied","Data":"728400baf3da2e3b0ab588ab57cf9de25f3c39c8d964a02160d446779645affd"} Oct 31 22:50:51.349078 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.348706 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"4823cb70f5c5f9288678441dce23411b86f53cc1b1532ceb4aa6c0ebc7967493"} Oct 31 22:50:51.350792 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.350765 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" event={"ID":"bc184cbe-a1dd-439c-a678-c3fe086ef90d","Type":"ContainerStarted","Data":"b2df38b0c7a0f71b76eb30e3cea24eb62b933fd59079b81ee9a0448d52fd8a30"} Oct 31 22:50:51.350917 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.350813 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" event={"ID":"bc184cbe-a1dd-439c-a678-c3fe086ef90d","Type":"ContainerStarted","Data":"728fe1f655788caf94c7703dffb8e1ab98243b257a037c43eb58724bfed712a6"} Oct 31 22:50:51.350917 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.350844 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" event={"ID":"bc184cbe-a1dd-439c-a678-c3fe086ef90d","Type":"ContainerStarted","Data":"6f744b0eb635c0aeb186b6703c85e474e9549687cf8259bd831129084631b4a1"} Oct 31 22:50:51.409777 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:51.409734 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dc4c664b9-xj859" podStartSLOduration=1.672627056 podStartE2EDuration="3.409721389s" totalImagesPullingTime="1.737094333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:51.409107566 +0000 UTC m=+29.391849943" watchObservedRunningTime="2026-10-31 22:50:51.409721389 +0000 UTC m=+29.392463760" Oct 31 22:50:52.139589 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.139559 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ec9d28-478d-45f2-82f5-8197104db046" path="/var/lib/kubelet/pods/72ec9d28-478d-45f2-82f5-8197104db046/volumes" Oct 31 22:50:52.359224 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359184 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"07b2007dbbeb1de09a0f1b63d55faa69ba67e5b4683196396217d0df2ccbd5ef"} Oct 31 22:50:52.359650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359254 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"03f132a79758df1953cc11d4689b4c25aae6964d5b14b53e0d7684e354b90d45"} Oct 31 22:50:52.359650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359287 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"21c60b18f3d50b1648e9761f9af810dd7bce5413e104894905e1d71b845da042"} Oct 31 22:50:52.359650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359313 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"c07caaf14ba158fbfa3b39853558959dbfeedb804b2193bfb22e259daa94013b"} Oct 31 22:50:52.359650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359348 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"2bf33fea95b2835a2ca99030cf2146acceaeabdd50051e9f33986d7be5e984ca"} Oct 31 22:50:52.359650 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.359378 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8324d2f9-f4bd-451a-925b-b6e9657b46d2","Type":"ContainerStarted","Data":"ae7f5c62265e539ad2dfc489d9211eba7e97f75ca14431ab5da2bfb1b7774ba4"} Oct 31 22:50:52.394943 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.394826 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.373535781 podStartE2EDuration="2.394807762s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.271981ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:52.394086999 +0000 UTC m=+30.376829379" watchObservedRunningTime="2026-10-31 22:50:52.394807762 +0000 UTC m=+30.377550143" Oct 31 22:50:52.943440 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:52.943406 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zsmmj" Oct 31 22:50:55.751239 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:50:55.751185 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:51:02.817379 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:02.817342 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:51:14.887249 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:51:14.887183 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:51:27.841103 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:27.841035 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d6f5dbdb9-6wwq7" podUID="a737f063-f6c6-4120-bd76-2b666dd9b154" containerName="console" containerID="cri-o://dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9" gracePeriod=15 Oct 31 22:51:28.081565 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.081543 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:51:28.126689 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126625 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.126689 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126658 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.126876 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126695 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.126876 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126820 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.127005 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126872 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.127005 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.126910 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.127115 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127013 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.127115 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127055 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") pod \"a737f063-f6c6-4120-bd76-2b666dd9b154\" (UID: \"a737f063-f6c6-4120-bd76-2b666dd9b154\") " Oct 31 22:51:28.127218 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127131 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:51:28.127273 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127234 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:51:28.127327 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127263 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:51:28.127327 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127294 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:51:28.127449 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127429 7074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-trusted-ca-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.127502 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127450 7074 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-console-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.127502 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127459 7074 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-oauth-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.127502 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127469 7074 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a737f063-f6c6-4120-bd76-2b666dd9b154-tmp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.127502 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.127453 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:51:28.129135 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.129111 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:51:28.129308 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.129294 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:51:28.129412 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.129390 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2" pod "a737f063-f6c6-4120-bd76-2b666dd9b154" (UID: "a737f063-f6c6-4120-bd76-2b666dd9b154"). InnerVolumeSpecName "kube-api-access-qv2q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:51:28.228257 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.228218 7074 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-serving-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.228257 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.228246 7074 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a737f063-f6c6-4120-bd76-2b666dd9b154-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.228257 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.228258 7074 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a737f063-f6c6-4120-bd76-2b666dd9b154-console-oauth-config\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.228485 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.228275 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv2q2\" (UniqueName: \"kubernetes.io/projected/a737f063-f6c6-4120-bd76-2b666dd9b154-kube-api-access-qv2q2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:51:28.365936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.365901 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:51:28.381408 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.381337 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:51:28.496511 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.496481 7074 generic.go:356] "Generic (PLEG): container finished" podID="a737f063-f6c6-4120-bd76-2b666dd9b154" containerID="dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9" exitCode=0 Oct 31 22:51:28.496687 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.496529 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d6f5dbdb9-6wwq7" Oct 31 22:51:28.496687 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.496581 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6f5dbdb9-6wwq7" event={"ID":"a737f063-f6c6-4120-bd76-2b666dd9b154","Type":"ContainerDied","Data":"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9"} Oct 31 22:51:28.496687 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.496634 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d6f5dbdb9-6wwq7" event={"ID":"a737f063-f6c6-4120-bd76-2b666dd9b154","Type":"ContainerDied","Data":"3304a7aa837fd7e9b62a0c3f0617bb272dbd97f11e890ceffb1079380a27ab6a"} Oct 31 22:51:28.496687 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.496654 7074 scope.go:122] "RemoveContainer" containerID="dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9" Oct 31 22:51:28.506117 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.506100 7074 scope.go:122] "RemoveContainer" containerID="dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9" Oct 31 22:51:28.506899 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:51:28.506870 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9\": container with ID starting with dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9 not found: ID does not exist" containerID="dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9" Oct 31 22:51:28.507105 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.507077 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9"} err="failed to get container status \"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9\": rpc error: code = NotFound desc = could not find container \"dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9\": container with ID starting with dad52f6771e5b2ce778dadda4c739da97687792d84b9a5b6988e7925d285dad9 not found: ID does not exist" Oct 31 22:51:28.515318 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.515299 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 22:51:28.515549 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.515531 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:51:28.521413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:28.521395 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d6f5dbdb9-6wwq7"] Oct 31 22:51:30.139304 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:51:30.139271 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a737f063-f6c6-4120-bd76-2b666dd9b154" path="/var/lib/kubelet/pods/a737f063-f6c6-4120-bd76-2b666dd9b154/volumes" Oct 31 22:51:44.886961 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:51:44.886922 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:52:14.887941 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:52:14.887855 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:52:44.886200 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:52:44.886165 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:53:13.466979 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.466942 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml"] Oct 31 22:53:13.467428 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.467286 7074 cpu_manager.go:414] "containerMap: removing container" podUID="a737f063-f6c6-4120-bd76-2b666dd9b154" containerName="console" Oct 31 22:53:13.467428 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.467298 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a737f063-f6c6-4120-bd76-2b666dd9b154" containerName="console" Oct 31 22:53:13.467428 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.467366 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a737f063-f6c6-4120-bd76-2b666dd9b154" containerName="console" Oct 31 22:53:13.470358 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.470338 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.475461 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.475422 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-f4jpz\"" Oct 31 22:53:13.475588 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.475527 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 22:53:13.475588 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.475586 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 22:53:13.480351 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.480326 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml"] Oct 31 22:53:13.528368 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.528336 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.528541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.528387 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8z5w\" (UniqueName: \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.528541 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.528496 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.532974 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.532949 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657"] Oct 31 22:53:13.539187 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.539161 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.546448 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.546423 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-dcc46\"" Oct 31 22:53:13.547361 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.547340 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657"] Oct 31 22:53:13.628441 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.628413 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h"] Oct 31 22:53:13.629893 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.629865 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8z5w\" (UniqueName: \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.630057 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.629900 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.630057 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.629931 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkd6h\" (UniqueName: \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.630156 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.630093 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.630156 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.630138 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.630265 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.630177 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.630487 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.630469 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.630566 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.630511 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.631948 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.631935 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.635625 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.635606 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lfrbn\"" Oct 31 22:53:13.642153 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.642130 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h"] Oct 31 22:53:13.642719 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.642696 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8z5w\" (UniqueName: \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.731085 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.730999 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.731085 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731054 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.731275 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731112 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.731275 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731150 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5czh\" (UniqueName: \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.731275 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731176 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkd6h\" (UniqueName: \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.731275 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731198 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.731482 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731458 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.731521 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.731488 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.744150 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.744116 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkd6h\" (UniqueName: \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.786921 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.786883 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:13.832632 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.832597 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5czh\" (UniqueName: \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.832805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.832641 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.832805 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.832701 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.833172 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.833150 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.833172 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.833162 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.846136 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.846102 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5czh\" (UniqueName: \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:13.856018 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.855793 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:13.948231 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:13.948199 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:14.061382 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.061347 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml"] Oct 31 22:53:14.065124 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:53:14.065100 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16727c5_96f4_47d7_98cd_23bb0b28c55e.slice/crio-17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941 WatchSource:0}: Error finding container 17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941: Status 404 returned error can't find the container with id 17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941 Oct 31 22:53:14.132427 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.132284 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657"] Oct 31 22:53:14.135044 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:53:14.135012 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181c13bd_37c9_46c5_8e65_8ba956848849.slice/crio-44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3 WatchSource:0}: Error finding container 44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3: Status 404 returned error can't find the container with id 44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3 Oct 31 22:53:14.220221 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.220191 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h"] Oct 31 22:53:14.223340 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:53:14.223307 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28096b_beb3_4834_8d30_278f2ccea2b9.slice/crio-4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd WatchSource:0}: Error finding container 4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd: Status 404 returned error can't find the container with id 4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd Oct 31 22:53:14.891936 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.891861 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" event={"ID":"7a28096b-beb3-4834-8d30-278f2ccea2b9","Type":"ContainerStarted","Data":"4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd"} Oct 31 22:53:14.896005 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.895902 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" event={"ID":"181c13bd-37c9-46c5-8e65-8ba956848849","Type":"ContainerStarted","Data":"44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3"} Oct 31 22:53:14.898954 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:14.898592 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" event={"ID":"e16727c5-96f4-47d7-98cd-23bb0b28c55e","Type":"ContainerStarted","Data":"17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941"} Oct 31 22:53:14.904969 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:53:14.903808 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:53:16.424402 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.424361 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq"] Oct 31 22:53:16.430278 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.430253 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.438984 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.438945 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-pjsld\"" Oct 31 22:53:16.452734 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.452672 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq"] Oct 31 22:53:16.453389 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.453224 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 22:53:16.453389 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.453300 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 22:53:16.453389 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.453357 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 22:53:16.559218 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559178 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559472 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559228 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mbz\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-kube-api-access-c5mbz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559472 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559322 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559472 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559349 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559472 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559380 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559472 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559409 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559758 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559587 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559758 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559622 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.559758 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.559642 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.660676 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660636 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.660872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660708 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.660872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660737 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.660872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660797 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.660872 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660824 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mbz\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-kube-api-access-c5mbz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661121 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660873 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661121 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660901 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661121 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660937 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661121 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.660966 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661903 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.661880 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.661903 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.661896 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.662290 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.662260 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.662555 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.662493 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.662794 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.662768 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.668165 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.668138 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.675600 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.675462 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mbz\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-kube-api-access-c5mbz\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.677031 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.676980 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.678671 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.678612 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9228d705-aae1-4adc-9fe5-9ce1e18afabc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq\" (UID: \"9228d705-aae1-4adc-9fe5-9ce1e18afabc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:16.755517 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:16.755482 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:17.071143 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:17.071112 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq"] Oct 31 22:53:17.917402 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:17.917341 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" event={"ID":"9228d705-aae1-4adc-9fe5-9ce1e18afabc","Type":"ContainerStarted","Data":"051df859fc20ba3de0a765470f94c559cea4c5011281cd27cb6ec6f926bcf973"} Oct 31 22:53:21.386601 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:21.386559 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 22:53:21.386920 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:21.386674 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 22:53:21.386920 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:21.386720 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 22:53:21.944124 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:21.944047 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" event={"ID":"9228d705-aae1-4adc-9fe5-9ce1e18afabc","Type":"ContainerStarted","Data":"e810dcd70fb339d122381143f78ad177d322c6d0007eefe0e9a6978f21072e62"} Oct 31 22:53:21.972537 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:21.972480 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" podStartSLOduration=1.6660337539999999 podStartE2EDuration="5.972461235s" totalImagesPullingTime="4.306427481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:21.970416518 +0000 UTC m=+179.953158897" watchObservedRunningTime="2026-10-31 22:53:21.972461235 +0000 UTC m=+179.955203614" Oct 31 22:53:22.756674 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.756635 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:22.761462 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.761438 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:22.948824 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.948743 7074 generic.go:356] "Generic (PLEG): container finished" podID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerID="ae6d25fbaf8d2d21cb6e85f291be96ca21232fe1343a7277c518463667351c22" exitCode=0 Oct 31 22:53:22.948974 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.948822 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" event={"ID":"7a28096b-beb3-4834-8d30-278f2ccea2b9","Type":"ContainerDied","Data":"ae6d25fbaf8d2d21cb6e85f291be96ca21232fe1343a7277c518463667351c22"} Oct 31 22:53:22.950448 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.950423 7074 generic.go:356] "Generic (PLEG): container finished" podID="181c13bd-37c9-46c5-8e65-8ba956848849" containerID="528c8986eed037ad7a64dfbb79b89532cf150752247ea7b09e9db53a33fc15ce" exitCode=0 Oct 31 22:53:22.950560 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.950486 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" event={"ID":"181c13bd-37c9-46c5-8e65-8ba956848849","Type":"ContainerDied","Data":"528c8986eed037ad7a64dfbb79b89532cf150752247ea7b09e9db53a33fc15ce"} Oct 31 22:53:22.952179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.952158 7074 generic.go:356] "Generic (PLEG): container finished" podID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerID="184d27354ec8b81b8c80afe6386da36733846145be7187d98878214161682d00" exitCode=0 Oct 31 22:53:22.952282 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.952238 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" event={"ID":"e16727c5-96f4-47d7-98cd-23bb0b28c55e","Type":"ContainerDied","Data":"184d27354ec8b81b8c80afe6386da36733846145be7187d98878214161682d00"} Oct 31 22:53:22.952592 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.952568 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:22.953668 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:22.953638 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq" Oct 31 22:53:24.961311 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.961277 7074 generic.go:356] "Generic (PLEG): container finished" podID="181c13bd-37c9-46c5-8e65-8ba956848849" containerID="40dbda1f6e90b5a239d6fbc2345a62b90615d0427dea395976904f017821cf07" exitCode=0 Oct 31 22:53:24.961787 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.961366 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" event={"ID":"181c13bd-37c9-46c5-8e65-8ba956848849","Type":"ContainerDied","Data":"40dbda1f6e90b5a239d6fbc2345a62b90615d0427dea395976904f017821cf07"} Oct 31 22:53:24.963284 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.963244 7074 generic.go:356] "Generic (PLEG): container finished" podID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerID="c54c3142d98fc79a5a11537464b62ca64fa92405812f1d97fa89bd9f65984dd4" exitCode=0 Oct 31 22:53:24.963423 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.963325 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" event={"ID":"e16727c5-96f4-47d7-98cd-23bb0b28c55e","Type":"ContainerDied","Data":"c54c3142d98fc79a5a11537464b62ca64fa92405812f1d97fa89bd9f65984dd4"} Oct 31 22:53:24.965300 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.965278 7074 generic.go:356] "Generic (PLEG): container finished" podID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerID="3b34648ab7c1e669378871d0f410b156783b079252c228fb2e5096871c8b5095" exitCode=0 Oct 31 22:53:24.965406 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:24.965331 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" event={"ID":"7a28096b-beb3-4834-8d30-278f2ccea2b9","Type":"ContainerDied","Data":"3b34648ab7c1e669378871d0f410b156783b079252c228fb2e5096871c8b5095"} Oct 31 22:53:32.922624 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:53:32.922573 7074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28096b_beb3_4834_8d30_278f2ccea2b9.slice/crio-conmon-87d227a2fc6252b9d3de6931935f9680df89f0e64889b9e7e73f21fe101e2a61.scope\": RecentStats: unable to find data in memory cache]" Oct 31 22:53:33.011325 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:33.011300 7074 generic.go:356] "Generic (PLEG): container finished" podID="181c13bd-37c9-46c5-8e65-8ba956848849" containerID="620be56edd5fa7fbc28e6dc4dffd32dfa374e940f3986a49f765f23dc637d8d2" exitCode=0 Oct 31 22:53:33.011433 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:33.011379 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" event={"ID":"181c13bd-37c9-46c5-8e65-8ba956848849","Type":"ContainerDied","Data":"620be56edd5fa7fbc28e6dc4dffd32dfa374e940f3986a49f765f23dc637d8d2"} Oct 31 22:53:33.013423 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:33.013403 7074 generic.go:356] "Generic (PLEG): container finished" podID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerID="87d227a2fc6252b9d3de6931935f9680df89f0e64889b9e7e73f21fe101e2a61" exitCode=0 Oct 31 22:53:33.013508 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:33.013430 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" event={"ID":"7a28096b-beb3-4834-8d30-278f2ccea2b9","Type":"ContainerDied","Data":"87d227a2fc6252b9d3de6931935f9680df89f0e64889b9e7e73f21fe101e2a61"} Oct 31 22:53:34.019534 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.019438 7074 generic.go:356] "Generic (PLEG): container finished" podID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerID="8c4408d15d805e8fa2f6c3ee5291141375a734bde1bc511d256a6d9c2a87e30e" exitCode=0 Oct 31 22:53:34.019893 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.019523 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" event={"ID":"e16727c5-96f4-47d7-98cd-23bb0b28c55e","Type":"ContainerDied","Data":"8c4408d15d805e8fa2f6c3ee5291141375a734bde1bc511d256a6d9c2a87e30e"} Oct 31 22:53:34.148774 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.148747 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:34.179282 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.179255 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:34.255392 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.255360 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\" (UniqueName: \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\") pod \"7a28096b-beb3-4834-8d30-278f2ccea2b9\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " Oct 31 22:53:34.255540 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.255468 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\") pod \"7a28096b-beb3-4834-8d30-278f2ccea2b9\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " Oct 31 22:53:34.255540 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.255498 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\") pod \"7a28096b-beb3-4834-8d30-278f2ccea2b9\" (UID: \"7a28096b-beb3-4834-8d30-278f2ccea2b9\") " Oct 31 22:53:34.256025 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.255979 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle" pod "7a28096b-beb3-4834-8d30-278f2ccea2b9" (UID: "7a28096b-beb3-4834-8d30-278f2ccea2b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.257548 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.257526 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh" pod "7a28096b-beb3-4834-8d30-278f2ccea2b9" (UID: "7a28096b-beb3-4834-8d30-278f2ccea2b9"). InnerVolumeSpecName "kube-api-access-t5czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:34.260179 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.260158 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util" pod "7a28096b-beb3-4834-8d30-278f2ccea2b9" (UID: "7a28096b-beb3-4834-8d30-278f2ccea2b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.356695 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.356612 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\") pod \"181c13bd-37c9-46c5-8e65-8ba956848849\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " Oct 31 22:53:34.356695 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.356696 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\") pod \"181c13bd-37c9-46c5-8e65-8ba956848849\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " Oct 31 22:53:34.356867 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.356742 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\" (UniqueName: \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\") pod \"181c13bd-37c9-46c5-8e65-8ba956848849\" (UID: \"181c13bd-37c9-46c5-8e65-8ba956848849\") " Oct 31 22:53:34.357029 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.357012 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5czh\" (UniqueName: \"kubernetes.io/projected/7a28096b-beb3-4834-8d30-278f2ccea2b9-kube-api-access-t5czh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.357093 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.357035 7074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-util\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.357093 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.357049 7074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a28096b-beb3-4834-8d30-278f2ccea2b9-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.357182 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.357159 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle" pod "181c13bd-37c9-46c5-8e65-8ba956848849" (UID: "181c13bd-37c9-46c5-8e65-8ba956848849"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.358767 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.358744 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h" pod "181c13bd-37c9-46c5-8e65-8ba956848849" (UID: "181c13bd-37c9-46c5-8e65-8ba956848849"). InnerVolumeSpecName "kube-api-access-rkd6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:34.361018 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.360976 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util" pod "181c13bd-37c9-46c5-8e65-8ba956848849" (UID: "181c13bd-37c9-46c5-8e65-8ba956848849"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.457834 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.457792 7074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-util\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.457834 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.457832 7074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/181c13bd-37c9-46c5-8e65-8ba956848849-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.457834 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:34.457843 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkd6h\" (UniqueName: \"kubernetes.io/projected/181c13bd-37c9-46c5-8e65-8ba956848849-kube-api-access-rkd6h\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:35.025421 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.025330 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" event={"ID":"7a28096b-beb3-4834-8d30-278f2ccea2b9","Type":"ContainerDied","Data":"4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd"} Oct 31 22:53:35.025421 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.025350 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767j5h" Oct 31 22:53:35.025421 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.025386 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4edc4ca19b06fe1004a64552342c14f8b4e5e49369e3faadc57c4bcf7c8373cd" Oct 31 22:53:35.027424 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.027404 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" Oct 31 22:53:35.027554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.027401 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6m657" event={"ID":"181c13bd-37c9-46c5-8e65-8ba956848849","Type":"ContainerDied","Data":"44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3"} Oct 31 22:53:35.027554 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.027521 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c1877f7f78b9de5ff66e76c71482eca15afd754b2ca46f7188a37135b0fcb3" Oct 31 22:53:35.147846 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.147821 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:35.162359 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.162337 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\") pod \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " Oct 31 22:53:35.162492 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.162368 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\" (UniqueName: \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\") pod \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " Oct 31 22:53:35.162492 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.162409 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\") pod \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\" (UID: \"e16727c5-96f4-47d7-98cd-23bb0b28c55e\") " Oct 31 22:53:35.163053 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.163032 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle" pod "e16727c5-96f4-47d7-98cd-23bb0b28c55e" (UID: "e16727c5-96f4-47d7-98cd-23bb0b28c55e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:35.164325 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.164306 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w" pod "e16727c5-96f4-47d7-98cd-23bb0b28c55e" (UID: "e16727c5-96f4-47d7-98cd-23bb0b28c55e"). InnerVolumeSpecName "kube-api-access-k8z5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:35.166315 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.166293 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util" pod "e16727c5-96f4-47d7-98cd-23bb0b28c55e" (UID: "e16727c5-96f4-47d7-98cd-23bb0b28c55e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:35.263629 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.263597 7074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-bundle\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:35.263629 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.263621 7074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e16727c5-96f4-47d7-98cd-23bb0b28c55e-util\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:35.263629 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:35.263631 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8z5w\" (UniqueName: \"kubernetes.io/projected/e16727c5-96f4-47d7-98cd-23bb0b28c55e-kube-api-access-k8z5w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.032882 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:36.032852 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" Oct 31 22:53:36.032882 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:36.032868 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax7cml" event={"ID":"e16727c5-96f4-47d7-98cd-23bb0b28c55e","Type":"ContainerDied","Data":"17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941"} Oct 31 22:53:36.033303 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:36.032942 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e36f5a294fcb27036472e5606046b6f38f88c735ca4bd6dd4fe8b9be0b3941" Oct 31 22:53:38.553263 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553234 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8"] Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553589 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="pull" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553601 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="pull" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553623 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553628 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553635 7074 cpu_manager.go:414] "containerMap: removing container" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="pull" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553642 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="pull" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553652 7074 cpu_manager.go:414] "containerMap: removing container" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553659 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553669 7074 cpu_manager.go:414] "containerMap: removing container" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="extract" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553674 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="extract" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553685 7074 cpu_manager.go:414] "containerMap: removing container" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553689 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="util" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553701 7074 cpu_manager.go:414] "containerMap: removing container" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="extract" Oct 31 22:53:38.553723 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553705 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="extract" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553739 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="extract" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553744 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="extract" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553751 7074 cpu_manager.go:414] "containerMap: removing container" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="pull" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553755 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="pull" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553816 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a28096b-beb3-4834-8d30-278f2ccea2b9" containerName="extract" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553827 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="181c13bd-37c9-46c5-8e65-8ba956848849" containerName="extract" Oct 31 22:53:38.554209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.553832 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16727c5-96f4-47d7-98cd-23bb0b28c55e" containerName="extract" Oct 31 22:53:38.607431 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.607401 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8"] Oct 31 22:53:38.607576 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.607532 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.626560 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.626541 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 22:53:38.626956 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.626941 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 22:53:38.627782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.627766 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 22:53:38.628010 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.627977 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 22:53:38.628082 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.628074 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hg7q6\"" Oct 31 22:53:38.628163 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.628151 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 22:53:38.687116 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687086 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.687250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687128 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.687250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687199 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.687250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687226 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.687250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687247 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.687383 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.687351 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4q8\" (UniqueName: \"kubernetes.io/projected/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-kube-api-access-xq4q8\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788383 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788354 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4q8\" (UniqueName: \"kubernetes.io/projected/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-kube-api-access-xq4q8\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788550 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788397 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788550 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788445 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788550 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788480 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788550 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788507 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788550 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788534 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.788856 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.788832 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.791013 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.790972 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.791147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.791011 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.791147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.791113 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.791147 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.791122 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.801110 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.801091 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4q8\" (UniqueName: \"kubernetes.io/projected/38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9-kube-api-access-xq4q8\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-wkct8\" (UID: \"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:38.955043 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:38.954949 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:39.277398 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:39.277371 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8"] Oct 31 22:53:39.279507 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:53:39.279478 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cdaf26_d9f6_44e5_a1f2_4f0b87507eb9.slice/crio-43cc62bdac517c9dadf5377277c944965988d258de2b1d9b56bdf46061b2ad21 WatchSource:0}: Error finding container 43cc62bdac517c9dadf5377277c944965988d258de2b1d9b56bdf46061b2ad21: Status 404 returned error can't find the container with id 43cc62bdac517c9dadf5377277c944965988d258de2b1d9b56bdf46061b2ad21 Oct 31 22:53:40.052343 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:40.052305 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" event={"ID":"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9","Type":"ContainerStarted","Data":"43cc62bdac517c9dadf5377277c944965988d258de2b1d9b56bdf46061b2ad21"} Oct 31 22:53:43.067597 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.067564 7074 generic.go:356] "Generic (PLEG): container finished" podID="38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9" containerID="909bdbc5321f6bed4931034276e37d2f65741b15dc1d177e556fa5cd4b71b4a3" exitCode=0 Oct 31 22:53:43.067983 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.067648 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" event={"ID":"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9","Type":"ContainerDied","Data":"909bdbc5321f6bed4931034276e37d2f65741b15dc1d177e556fa5cd4b71b4a3"} Oct 31 22:53:43.287909 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.287870 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v"] Oct 31 22:53:43.290725 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.290688 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.301005 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.300642 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-rpktb\"" Oct 31 22:53:43.302080 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.301241 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 22:53:43.302080 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.301314 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:53:43.302080 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.301872 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v"] Oct 31 22:53:43.331223 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.331159 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030efd79-76e3-42bd-a161-e154dca00b6b-tmp\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.331223 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.331216 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmm9w\" (UniqueName: \"kubernetes.io/projected/030efd79-76e3-42bd-a161-e154dca00b6b-kube-api-access-hmm9w\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.431705 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.431673 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030efd79-76e3-42bd-a161-e154dca00b6b-tmp\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.431870 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.431741 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmm9w\" (UniqueName: \"kubernetes.io/projected/030efd79-76e3-42bd-a161-e154dca00b6b-kube-api-access-hmm9w\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.432098 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.432078 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030efd79-76e3-42bd-a161-e154dca00b6b-tmp\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.444328 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.444305 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmm9w\" (UniqueName: \"kubernetes.io/projected/030efd79-76e3-42bd-a161-e154dca00b6b-kube-api-access-hmm9w\") pod \"jobset-operator-674cd59dc5-6pj9v\" (UID: \"030efd79-76e3-42bd-a161-e154dca00b6b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.619576 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.619552 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" Oct 31 22:53:43.922336 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:43.922262 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v"] Oct 31 22:53:43.925711 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:53:43.925684 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod030efd79_76e3_42bd_a161_e154dca00b6b.slice/crio-e0a35a9877f9ce47d27578adfa4211ce3b126c6a6d18669aa6e4b7d1382d971c WatchSource:0}: Error finding container e0a35a9877f9ce47d27578adfa4211ce3b126c6a6d18669aa6e4b7d1382d971c: Status 404 returned error can't find the container with id e0a35a9877f9ce47d27578adfa4211ce3b126c6a6d18669aa6e4b7d1382d971c Oct 31 22:53:44.073479 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:44.073439 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" event={"ID":"38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9","Type":"ContainerStarted","Data":"372a20eafca080a772826d7a56c3726cb488c3adb07598c0980e272625d44b76"} Oct 31 22:53:44.073932 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:44.073595 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:53:44.074841 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:44.074820 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" event={"ID":"030efd79-76e3-42bd-a161-e154dca00b6b","Type":"ContainerStarted","Data":"e0a35a9877f9ce47d27578adfa4211ce3b126c6a6d18669aa6e4b7d1382d971c"} Oct 31 22:53:44.117859 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:44.117816 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" podStartSLOduration=2.293219307 podStartE2EDuration="6.117804795s" totalImagesPullingTime="3.368185253s" totalInitContainerRuntime="456.400235ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:44.116208442 +0000 UTC m=+202.098950821" watchObservedRunningTime="2026-10-31 22:53:44.117804795 +0000 UTC m=+202.100547172" Oct 31 22:53:44.887912 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:53:44.887876 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:53:48.096714 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:48.096671 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" event={"ID":"030efd79-76e3-42bd-a161-e154dca00b6b","Type":"ContainerStarted","Data":"6aa3d0192d8b7ab0f2bc7669cdc200d2afea808459fa3fd2ddfd3c6954d74b06"} Oct 31 22:53:48.117851 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:48.117794 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6pj9v" podStartSLOduration=1.699632706 podStartE2EDuration="5.117779108s" totalImagesPullingTime="3.418146402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:48.115261345 +0000 UTC m=+206.098003723" watchObservedRunningTime="2026-10-31 22:53:48.117779108 +0000 UTC m=+206.100521486" Oct 31 22:53:55.082378 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:53:55.082349 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-wkct8" Oct 31 22:54:03.477628 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.477588 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dcdms"] Oct 31 22:54:03.481075 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.481053 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:03.486861 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.486836 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 22:54:03.488205 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.488178 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 22:54:03.488303 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.488282 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ndlzl\"" Oct 31 22:54:03.494951 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.494930 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dcdms"] Oct 31 22:54:03.610625 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.610592 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8hb\" (UniqueName: \"kubernetes.io/projected/395fc085-9125-4599-bf52-aa25d1f7583a-kube-api-access-nc8hb\") pod \"authorino-operator-7b7bb4bc77-dcdms\" (UID: \"395fc085-9125-4599-bf52-aa25d1f7583a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:03.711388 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.711354 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8hb\" (UniqueName: \"kubernetes.io/projected/395fc085-9125-4599-bf52-aa25d1f7583a-kube-api-access-nc8hb\") pod \"authorino-operator-7b7bb4bc77-dcdms\" (UID: \"395fc085-9125-4599-bf52-aa25d1f7583a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:03.730040 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.729951 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8hb\" (UniqueName: \"kubernetes.io/projected/395fc085-9125-4599-bf52-aa25d1f7583a-kube-api-access-nc8hb\") pod \"authorino-operator-7b7bb4bc77-dcdms\" (UID: \"395fc085-9125-4599-bf52-aa25d1f7583a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:03.794225 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:03.794189 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:04.068463 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:04.068440 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dcdms"] Oct 31 22:54:04.071085 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:54:04.071057 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395fc085_9125_4599_bf52_aa25d1f7583a.slice/crio-b70e8419cbc7a5f4b1c21eb4b2ad28c69351dc26c3f75e8489c90c60e1235216 WatchSource:0}: Error finding container b70e8419cbc7a5f4b1c21eb4b2ad28c69351dc26c3f75e8489c90c60e1235216: Status 404 returned error can't find the container with id b70e8419cbc7a5f4b1c21eb4b2ad28c69351dc26c3f75e8489c90c60e1235216 Oct 31 22:54:04.165129 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:04.165092 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" event={"ID":"395fc085-9125-4599-bf52-aa25d1f7583a","Type":"ContainerStarted","Data":"b70e8419cbc7a5f4b1c21eb4b2ad28c69351dc26c3f75e8489c90c60e1235216"} Oct 31 22:54:05.621609 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.621571 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq"] Oct 31 22:54:05.625098 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.625076 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.635094 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.635074 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 22:54:05.635206 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.635079 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 22:54:05.638233 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.638212 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lhgvp\"" Oct 31 22:54:05.638654 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.638629 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq"] Oct 31 22:54:05.736233 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.736196 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754d6\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-kube-api-access-754d6\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.736408 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.736271 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.837761 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.837724 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754d6\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-kube-api-access-754d6\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.837919 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.837785 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.856359 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.856331 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.861725 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.861702 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-754d6\" (UniqueName: \"kubernetes.io/projected/afa25ec3-46cb-4cf1-939f-083544cae59c-kube-api-access-754d6\") pod \"cert-manager-cainjector-5c57bbd4fc-p49jq\" (UID: \"afa25ec3-46cb-4cf1-939f-083544cae59c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:05.940138 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:05.940061 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" Oct 31 22:54:06.220250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:06.220180 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq"] Oct 31 22:54:06.224116 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:54:06.224088 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa25ec3_46cb_4cf1_939f_083544cae59c.slice/crio-5db1dfffa8b951c11901f0bbb8b64ecf1f49695c46fc414b921346a6a8cd5432 WatchSource:0}: Error finding container 5db1dfffa8b951c11901f0bbb8b64ecf1f49695c46fc414b921346a6a8cd5432: Status 404 returned error can't find the container with id 5db1dfffa8b951c11901f0bbb8b64ecf1f49695c46fc414b921346a6a8cd5432 Oct 31 22:54:07.180232 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:07.180190 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" event={"ID":"afa25ec3-46cb-4cf1-939f-083544cae59c","Type":"ContainerStarted","Data":"5db1dfffa8b951c11901f0bbb8b64ecf1f49695c46fc414b921346a6a8cd5432"} Oct 31 22:54:08.188400 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:08.188361 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" event={"ID":"395fc085-9125-4599-bf52-aa25d1f7583a","Type":"ContainerStarted","Data":"f6348ffb79e4fdfab74196bef676956e35a1e968e107c04aa34723e41993c7be"} Oct 31 22:54:08.188884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:08.188616 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:08.211524 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:08.211456 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" podStartSLOduration=1.99857788 podStartE2EDuration="5.211430641s" totalImagesPullingTime="3.212852761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:08.210275694 +0000 UTC m=+226.193018072" watchObservedRunningTime="2026-10-31 22:54:08.211430641 +0000 UTC m=+226.194173018" Oct 31 22:54:10.199255 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:10.199222 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" event={"ID":"afa25ec3-46cb-4cf1-939f-083544cae59c","Type":"ContainerStarted","Data":"1f8f6dffacbc1c75f63e822963b234cbf0c42c2f03a65284babcf3ae499fbf0c"} Oct 31 22:54:10.221070 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:10.221014 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p49jq" podStartSLOduration=1.389950812 podStartE2EDuration="5.220975218s" totalImagesPullingTime="3.831024406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:10.220731675 +0000 UTC m=+228.203474054" watchObservedRunningTime="2026-10-31 22:54:10.220975218 +0000 UTC m=+228.203717596" Oct 31 22:54:12.309711 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.309681 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-9g96x"] Oct 31 22:54:12.313092 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.313065 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.322584 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.322542 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9pfzm\"" Oct 31 22:54:12.324157 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.324132 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9g96x"] Oct 31 22:54:12.396841 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.396810 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-bound-sa-token\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.397058 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.396885 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsct\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-kube-api-access-trsct\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.498387 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.498354 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsct\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-kube-api-access-trsct\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.498533 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.498470 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-bound-sa-token\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.510978 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.510946 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-bound-sa-token\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.511306 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.511282 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsct\" (UniqueName: \"kubernetes.io/projected/75c7b50a-8eda-47a4-aa5b-ec7ddf353189-kube-api-access-trsct\") pod \"cert-manager-597b95cb84-9g96x\" (UID: \"75c7b50a-8eda-47a4-aa5b-ec7ddf353189\") " pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.625780 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.625704 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9g96x" Oct 31 22:54:12.904459 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:12.904370 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9g96x"] Oct 31 22:54:12.907301 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:54:12.907266 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c7b50a_8eda_47a4_aa5b_ec7ddf353189.slice/crio-db9428d3d86d7422435a6df70ea1f88748b2573cdab473dd8157b3c3a81f8ad2 WatchSource:0}: Error finding container db9428d3d86d7422435a6df70ea1f88748b2573cdab473dd8157b3c3a81f8ad2: Status 404 returned error can't find the container with id db9428d3d86d7422435a6df70ea1f88748b2573cdab473dd8157b3c3a81f8ad2 Oct 31 22:54:13.213632 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:13.213550 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9g96x" event={"ID":"75c7b50a-8eda-47a4-aa5b-ec7ddf353189","Type":"ContainerStarted","Data":"2ba811f7b30230be8b685e2e42a5e9bec26934204872d1fdbc4ad1c3f5ff4048"} Oct 31 22:54:13.213632 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:13.213599 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9g96x" event={"ID":"75c7b50a-8eda-47a4-aa5b-ec7ddf353189","Type":"ContainerStarted","Data":"db9428d3d86d7422435a6df70ea1f88748b2573cdab473dd8157b3c3a81f8ad2"} Oct 31 22:54:13.237317 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:13.237270 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-9g96x" podStartSLOduration=1.2372577200000001 podStartE2EDuration="1.23725772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:13.235503355 +0000 UTC m=+231.218245734" watchObservedRunningTime="2026-10-31 22:54:13.23725772 +0000 UTC m=+231.220000099" Oct 31 22:54:14.887875 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:54:14.887835 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:54:15.430503 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.430471 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg"] Oct 31 22:54:15.436251 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.436225 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.448065 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.448037 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg"] Oct 31 22:54:15.476498 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.476461 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 22:54:15.476742 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.476461 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 22:54:15.476742 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.476559 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7xzl9\"" Oct 31 22:54:15.534251 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.534206 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8a48-7955-498e-a9da-6030072373b6-tls-certs\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.534251 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.534262 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1af8a48-7955-498e-a9da-6030072373b6-kube-api-access-79qdg\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.534445 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.534375 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1af8a48-7955-498e-a9da-6030072373b6-tmp\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.635089 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.635053 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8a48-7955-498e-a9da-6030072373b6-tls-certs\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.635319 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.635109 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1af8a48-7955-498e-a9da-6030072373b6-kube-api-access-79qdg\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.635319 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.635169 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1af8a48-7955-498e-a9da-6030072373b6-tmp\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.637413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.637384 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1af8a48-7955-498e-a9da-6030072373b6-tmp\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.637605 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.637586 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8a48-7955-498e-a9da-6030072373b6-tls-certs\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.661030 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.660977 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1af8a48-7955-498e-a9da-6030072373b6-kube-api-access-79qdg\") pod \"kube-auth-proxy-689d5458bb-bxxtg\" (UID: \"f1af8a48-7955-498e-a9da-6030072373b6\") " pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:15.751037 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:15.750999 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" Oct 31 22:54:16.030150 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:16.029980 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg"] Oct 31 22:54:16.032674 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:54:16.032647 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1af8a48_7955_498e_a9da_6030072373b6.slice/crio-26def7d6011a3e6de195e5114c114c2e98de95c517dcae95709f2cdd2f36ed81 WatchSource:0}: Error finding container 26def7d6011a3e6de195e5114c114c2e98de95c517dcae95709f2cdd2f36ed81: Status 404 returned error can't find the container with id 26def7d6011a3e6de195e5114c114c2e98de95c517dcae95709f2cdd2f36ed81 Oct 31 22:54:16.228267 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:16.228231 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" event={"ID":"f1af8a48-7955-498e-a9da-6030072373b6","Type":"ContainerStarted","Data":"26def7d6011a3e6de195e5114c114c2e98de95c517dcae95709f2cdd2f36ed81"} Oct 31 22:54:19.196500 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:19.196463 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dcdms" Oct 31 22:54:19.247405 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:19.247365 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" event={"ID":"f1af8a48-7955-498e-a9da-6030072373b6","Type":"ContainerStarted","Data":"237a9f783ac2c5802dcc27ddb01aa923e604533bfb3270f2d88ddd0b0264eaeb"} Oct 31 22:54:19.274940 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:54:19.274672 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-689d5458bb-bxxtg" podStartSLOduration=1.863757982 podStartE2EDuration="4.274654356s" totalImagesPullingTime="2.410896374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:19.267413159 +0000 UTC m=+237.250155561" watchObservedRunningTime="2026-10-31 22:54:19.274654356 +0000 UTC m=+237.257396735" Oct 31 22:54:44.887164 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:54:44.887127 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:55:10.345010 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.344920 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xzrsr"] Oct 31 22:55:10.348301 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.348282 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.353970 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.353950 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 22:55:10.358095 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.356596 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 22:55:10.358209 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.358150 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4lsnq\"" Oct 31 22:55:10.358276 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.358159 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 22:55:10.370858 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.370834 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xzrsr"] Oct 31 22:55:10.396085 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.396046 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmkf2"] Oct 31 22:55:10.409739 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.409708 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.412878 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.412850 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmkf2"] Oct 31 22:55:10.456953 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.456923 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 22:55:10.456953 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.456923 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-srmt8\"" Oct 31 22:55:10.526012 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.525951 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9k6n\" (UniqueName: \"kubernetes.io/projected/fbcbb847-ae4b-43ce-bd43-9ab91203d732-kube-api-access-h9k6n\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.526012 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.526003 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-cert\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.526250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.526036 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnq7\" (UniqueName: \"kubernetes.io/projected/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-kube-api-access-8rnq7\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.526250 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.526172 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcbb847-ae4b-43ce-bd43-9ab91203d732-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.627398 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.627311 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9k6n\" (UniqueName: \"kubernetes.io/projected/fbcbb847-ae4b-43ce-bd43-9ab91203d732-kube-api-access-h9k6n\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.627398 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.627352 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-cert\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.627398 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.627371 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnq7\" (UniqueName: \"kubernetes.io/projected/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-kube-api-access-8rnq7\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.627656 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.627415 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcbb847-ae4b-43ce-bd43-9ab91203d732-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.629782 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.629759 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-cert\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.629845 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.629826 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcbb847-ae4b-43ce-bd43-9ab91203d732-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.652224 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.652191 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnq7\" (UniqueName: \"kubernetes.io/projected/ea06ed76-cafd-4c4d-8c0e-b687efd71a0a-kube-api-access-8rnq7\") pod \"kserve-controller-manager-776cc66d47-xzrsr\" (UID: \"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.652356 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.652292 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9k6n\" (UniqueName: \"kubernetes.io/projected/fbcbb847-ae4b-43ce-bd43-9ab91203d732-kube-api-access-h9k6n\") pod \"model-serving-api-75bd89fdfc-fmkf2\" (UID: \"fbcbb847-ae4b-43ce-bd43-9ab91203d732\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.662962 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.662937 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:10.724602 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.724575 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:10.938192 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:10.938159 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xzrsr"] Oct 31 22:55:10.940813 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:55:10.940785 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea06ed76_cafd_4c4d_8c0e_b687efd71a0a.slice/crio-141d6d837f718b1d7205244097acd9f6033337362268acc10d4da3048b113443 WatchSource:0}: Error finding container 141d6d837f718b1d7205244097acd9f6033337362268acc10d4da3048b113443: Status 404 returned error can't find the container with id 141d6d837f718b1d7205244097acd9f6033337362268acc10d4da3048b113443 Oct 31 22:55:11.008479 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:11.008448 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmkf2"] Oct 31 22:55:11.011198 ip-10-0-131-157 kubenswrapper[7074]: W1031 22:55:11.011173 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcbb847_ae4b_43ce_bd43_9ab91203d732.slice/crio-67dee8ac2f934870910b5433b6f7ec842b7b0782ed62f6a904ab70210b7fe62a WatchSource:0}: Error finding container 67dee8ac2f934870910b5433b6f7ec842b7b0782ed62f6a904ab70210b7fe62a: Status 404 returned error can't find the container with id 67dee8ac2f934870910b5433b6f7ec842b7b0782ed62f6a904ab70210b7fe62a Oct 31 22:55:11.501363 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:11.501318 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" event={"ID":"fbcbb847-ae4b-43ce-bd43-9ab91203d732","Type":"ContainerStarted","Data":"67dee8ac2f934870910b5433b6f7ec842b7b0782ed62f6a904ab70210b7fe62a"} Oct 31 22:55:11.502862 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:11.502835 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" event={"ID":"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a","Type":"ContainerStarted","Data":"141d6d837f718b1d7205244097acd9f6033337362268acc10d4da3048b113443"} Oct 31 22:55:13.515676 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.515634 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" event={"ID":"fbcbb847-ae4b-43ce-bd43-9ab91203d732","Type":"ContainerStarted","Data":"1c086f75d7591771368ad74542a329c195625c078cafe16f93596769adfe4836"} Oct 31 22:55:13.516146 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.515751 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:13.517138 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.517118 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" event={"ID":"ea06ed76-cafd-4c4d-8c0e-b687efd71a0a","Type":"ContainerStarted","Data":"6070109dfed34fe3a1141613df37be78cbb4b8b69e724361a6764f1389a24ddb"} Oct 31 22:55:13.517263 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.517247 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:13.541492 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.541441 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" podStartSLOduration=1.644869548 podStartE2EDuration="3.541427958s" totalImagesPullingTime="1.89655841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:55:13.540261817 +0000 UTC m=+291.523004196" watchObservedRunningTime="2026-10-31 22:55:13.541427958 +0000 UTC m=+291.524170336" Oct 31 22:55:13.564413 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:13.564363 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" podStartSLOduration=1.367404598 podStartE2EDuration="3.564350392s" totalImagesPullingTime="2.196945794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:55:13.564153269 +0000 UTC m=+291.546895649" watchObservedRunningTime="2026-10-31 22:55:13.564350392 +0000 UTC m=+291.547092770" Oct 31 22:55:14.887520 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:55:14.887476 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:55:22.176381 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:22.176352 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:55:22.177884 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:22.177850 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 22:55:24.526260 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:24.526230 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-fmkf2" Oct 31 22:55:44.528123 ip-10-0-131-157 kubenswrapper[7074]: I1031 22:55:44.528091 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xzrsr" Oct 31 22:55:44.887764 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:55:44.887672 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:56:14.887271 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:56:14.887231 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:56:44.887194 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:56:44.887152 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:57:14.886878 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:57:14.886837 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:57:44.886538 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:57:44.886501 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:58:14.886793 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:58:14.886715 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:58:44.888193 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:58:44.888149 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:59:14.886514 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:59:14.886474 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 22:59:44.886742 ip-10-0-131-157 kubenswrapper[7074]: E1031 22:59:44.886704 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:00:14.887257 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:00:14.887216 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:00:22.210321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:00:22.210291 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:00:22.212164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:00:22.212144 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:00:44.887357 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:00:44.887319 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:01:14.887584 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:01:14.887506 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:01:44.887156 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:01:44.887118 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:02:14.887120 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:02:14.887085 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:02:44.888256 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:02:44.888219 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:03:14.887102 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:03:14.887061 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:03:44.886482 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:03:44.886445 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:04:14.888320 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:04:14.888215 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:04:44.886267 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:04:44.886224 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:04:53.280579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.280539 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:04:53.284454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.284432 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.288493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.288457 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 23:04:53.288602 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.288531 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kjfr7\"" Oct 31 23:04:53.292932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.292909 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:04:53.316589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.316559 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4p9f\" (UniqueName: \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.316748 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.316668 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.385197 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.385168 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:04:53.417233 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.417196 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.417400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.417290 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4p9f\" (UniqueName: \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.417934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.417915 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.429490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.429467 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4p9f\" (UniqueName: \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\") pod \"limitador-limitador-74dfcfb894-chr2p\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.598796 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.598706 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:53.870849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.870824 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:04:53.873290 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:04:53.873261 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691a1e6c_7730_45d9_861d_5e348789440d.slice/crio-49bf9afa06ee9ad5d6b71cc6e1dfafa0d7608f520e8cb908aa96416098d3324f WatchSource:0}: Error finding container 49bf9afa06ee9ad5d6b71cc6e1dfafa0d7608f520e8cb908aa96416098d3324f: Status 404 returned error can't find the container with id 49bf9afa06ee9ad5d6b71cc6e1dfafa0d7608f520e8cb908aa96416098d3324f Oct 31 23:04:53.875265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:53.875251 7074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:04:54.076011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.075960 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:04:54.080859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.080836 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:54.087685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.087651 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:04:54.088684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.088646 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p44kh\"" Oct 31 23:04:54.123515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.123434 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zsl\" (UniqueName: \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\") pod \"authorino-57f7f8659f-8h6pg\" (UID: \"91ea2b43-5e40-4d99-9640-c103bde8e09a\") " pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:54.175780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.175746 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" event={"ID":"691a1e6c-7730-45d9-861d-5e348789440d","Type":"ContainerStarted","Data":"49bf9afa06ee9ad5d6b71cc6e1dfafa0d7608f520e8cb908aa96416098d3324f"} Oct 31 23:04:54.224086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.224039 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zsl\" (UniqueName: \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\") pod \"authorino-57f7f8659f-8h6pg\" (UID: \"91ea2b43-5e40-4d99-9640-c103bde8e09a\") " pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:54.237561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.237537 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zsl\" (UniqueName: \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\") pod \"authorino-57f7f8659f-8h6pg\" (UID: \"91ea2b43-5e40-4d99-9640-c103bde8e09a\") " pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:54.394019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.393924 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:54.665658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:54.665631 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:04:54.668149 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:04:54.668120 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ea2b43_5e40_4d99_9640_c103bde8e09a.slice/crio-4c15f7eda532653a4b1e6fb5aab244ab33656dfc9add7af150c1a5eff0cfd855 WatchSource:0}: Error finding container 4c15f7eda532653a4b1e6fb5aab244ab33656dfc9add7af150c1a5eff0cfd855: Status 404 returned error can't find the container with id 4c15f7eda532653a4b1e6fb5aab244ab33656dfc9add7af150c1a5eff0cfd855 Oct 31 23:04:55.183280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:55.183248 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" event={"ID":"91ea2b43-5e40-4d99-9640-c103bde8e09a","Type":"ContainerStarted","Data":"4c15f7eda532653a4b1e6fb5aab244ab33656dfc9add7af150c1a5eff0cfd855"} Oct 31 23:04:56.190889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:56.190850 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" event={"ID":"691a1e6c-7730-45d9-861d-5e348789440d","Type":"ContainerStarted","Data":"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020"} Oct 31 23:04:56.191337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:56.190976 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:04:56.211568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:56.211523 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" podStartSLOduration=1.038464849 podStartE2EDuration="3.211507661s" totalImagesPullingTime="2.173042812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:56.210071473 +0000 UTC m=+874.192813854" watchObservedRunningTime="2026-10-31 23:04:56.211507661 +0000 UTC m=+874.194250041" Oct 31 23:04:57.202925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:57.202869 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" event={"ID":"91ea2b43-5e40-4d99-9640-c103bde8e09a","Type":"ContainerStarted","Data":"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359"} Oct 31 23:04:57.219886 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:57.219834 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" podStartSLOduration=1.360858119 podStartE2EDuration="3.219817564s" totalImagesPullingTime="1.858959445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:57.219788104 +0000 UTC m=+875.202530486" watchObservedRunningTime="2026-10-31 23:04:57.219817564 +0000 UTC m=+875.202559943" Oct 31 23:04:57.361803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:57.361770 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:04:59.212146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:59.212103 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" podUID="91ea2b43-5e40-4d99-9640-c103bde8e09a" containerName="authorino" containerID="cri-o://7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359" gracePeriod=30 Oct 31 23:04:59.450848 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:59.450825 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:04:59.473050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:59.472977 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\" (UniqueName: \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\") pod \"91ea2b43-5e40-4d99-9640-c103bde8e09a\" (UID: \"91ea2b43-5e40-4d99-9640-c103bde8e09a\") " Oct 31 23:04:59.475130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:59.475097 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl" pod "91ea2b43-5e40-4d99-9640-c103bde8e09a" (UID: "91ea2b43-5e40-4d99-9640-c103bde8e09a"). InnerVolumeSpecName "kube-api-access-t9zsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:04:59.573728 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:04:59.573695 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zsl\" (UniqueName: \"kubernetes.io/projected/91ea2b43-5e40-4d99-9640-c103bde8e09a-kube-api-access-t9zsl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:00.218203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.218168 7074 generic.go:356] "Generic (PLEG): container finished" podID="91ea2b43-5e40-4d99-9640-c103bde8e09a" containerID="7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359" exitCode=0 Oct 31 23:05:00.218566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.218211 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" Oct 31 23:05:00.218566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.218254 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" event={"ID":"91ea2b43-5e40-4d99-9640-c103bde8e09a","Type":"ContainerDied","Data":"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359"} Oct 31 23:05:00.218566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.218300 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8h6pg" event={"ID":"91ea2b43-5e40-4d99-9640-c103bde8e09a","Type":"ContainerDied","Data":"4c15f7eda532653a4b1e6fb5aab244ab33656dfc9add7af150c1a5eff0cfd855"} Oct 31 23:05:00.218566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.218333 7074 scope.go:122] "RemoveContainer" containerID="7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359" Oct 31 23:05:00.226670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.226655 7074 scope.go:122] "RemoveContainer" containerID="7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359" Oct 31 23:05:00.226942 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:05:00.226923 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359\": container with ID starting with 7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359 not found: ID does not exist" containerID="7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359" Oct 31 23:05:00.226980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.226949 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359"} err="failed to get container status \"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359\": rpc error: code = NotFound desc = could not find container \"7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359\": container with ID starting with 7eee431e42639fb2de7d6454db69de3da9ad972b2f4ac0827e8f4017cc3c4359 not found: ID does not exist" Oct 31 23:05:00.235709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.235688 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:05:00.239955 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:00.239936 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8h6pg"] Oct 31 23:05:02.139710 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:02.139681 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ea2b43-5e40-4d99-9640-c103bde8e09a" path="/var/lib/kubelet/pods/91ea2b43-5e40-4d99-9640-c103bde8e09a/volumes" Oct 31 23:05:07.204788 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:07.204754 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:05:08.974633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:08.974600 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:05:08.975057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:08.974768 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" podUID="691a1e6c-7730-45d9-861d-5e348789440d" containerName="limitador" containerID="cri-o://8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020" gracePeriod=30 Oct 31 23:05:09.526827 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.526803 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:05:09.561019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.560936 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\" (UniqueName: \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\") pod \"691a1e6c-7730-45d9-861d-5e348789440d\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " Oct 31 23:05:09.561159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.561103 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\" (UniqueName: \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\") pod \"691a1e6c-7730-45d9-861d-5e348789440d\" (UID: \"691a1e6c-7730-45d9-861d-5e348789440d\") " Oct 31 23:05:09.561342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.561319 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file" pod "691a1e6c-7730-45d9-861d-5e348789440d" (UID: "691a1e6c-7730-45d9-861d-5e348789440d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:05:09.561431 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.561419 7074 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/691a1e6c-7730-45d9-861d-5e348789440d-config-file\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:09.563077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.563056 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f" pod "691a1e6c-7730-45d9-861d-5e348789440d" (UID: "691a1e6c-7730-45d9-861d-5e348789440d"). InnerVolumeSpecName "kube-api-access-f4p9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:05:09.661815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:09.661784 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4p9f\" (UniqueName: \"kubernetes.io/projected/691a1e6c-7730-45d9-861d-5e348789440d-kube-api-access-f4p9f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:10.269537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.269501 7074 generic.go:356] "Generic (PLEG): container finished" podID="691a1e6c-7730-45d9-861d-5e348789440d" containerID="8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020" exitCode=0 Oct 31 23:05:10.270035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.269553 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" event={"ID":"691a1e6c-7730-45d9-861d-5e348789440d","Type":"ContainerDied","Data":"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020"} Oct 31 23:05:10.270035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.269562 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" Oct 31 23:05:10.270035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.269605 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-chr2p" event={"ID":"691a1e6c-7730-45d9-861d-5e348789440d","Type":"ContainerDied","Data":"49bf9afa06ee9ad5d6b71cc6e1dfafa0d7608f520e8cb908aa96416098d3324f"} Oct 31 23:05:10.270035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.269635 7074 scope.go:122] "RemoveContainer" containerID="8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020" Oct 31 23:05:10.278221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.278195 7074 scope.go:122] "RemoveContainer" containerID="8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020" Oct 31 23:05:10.278516 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:05:10.278498 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020\": container with ID starting with 8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020 not found: ID does not exist" containerID="8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020" Oct 31 23:05:10.278597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.278524 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020"} err="failed to get container status \"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020\": rpc error: code = NotFound desc = could not find container \"8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020\": container with ID starting with 8bf691bc5ee3c6f3ae0373b74b60518ed21558505c4a585d044fa10c54342020 not found: ID does not exist" Oct 31 23:05:10.290576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.290556 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:05:10.300211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:10.300189 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-chr2p"] Oct 31 23:05:11.946006 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.945967 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q"] Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946329 7074 cpu_manager.go:414] "containerMap: removing container" podUID="691a1e6c-7730-45d9-861d-5e348789440d" containerName="limitador" Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946341 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691a1e6c-7730-45d9-861d-5e348789440d" containerName="limitador" Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946368 7074 cpu_manager.go:414] "containerMap: removing container" podUID="91ea2b43-5e40-4d99-9640-c103bde8e09a" containerName="authorino" Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946373 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ea2b43-5e40-4d99-9640-c103bde8e09a" containerName="authorino" Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946440 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="691a1e6c-7730-45d9-861d-5e348789440d" containerName="limitador" Oct 31 23:05:11.946524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.946454 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ea2b43-5e40-4d99-9640-c103bde8e09a" containerName="authorino" Oct 31 23:05:11.949436 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.949420 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:11.953502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.953481 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 23:05:11.953502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.953481 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:05:11.955054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.955035 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-88q69\"" Oct 31 23:05:11.956758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.956466 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:05:11.958526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:11.958500 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q"] Oct 31 23:05:12.082713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.082677 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0f23d37-4482-4829-8b98-93819e55af01-data\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.082713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.082713 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7h8x\" (UniqueName: \"kubernetes.io/projected/d0f23d37-4482-4829-8b98-93819e55af01-kube-api-access-w7h8x\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.141710 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.141672 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691a1e6c-7730-45d9-861d-5e348789440d" path="/var/lib/kubelet/pods/691a1e6c-7730-45d9-861d-5e348789440d/volumes" Oct 31 23:05:12.183845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.183813 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0f23d37-4482-4829-8b98-93819e55af01-data\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.184057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.183852 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7h8x\" (UniqueName: \"kubernetes.io/projected/d0f23d37-4482-4829-8b98-93819e55af01-kube-api-access-w7h8x\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.184217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.184198 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0f23d37-4482-4829-8b98-93819e55af01-data\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.196865 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.196806 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7h8x\" (UniqueName: \"kubernetes.io/projected/d0f23d37-4482-4829-8b98-93819e55af01-kube-api-access-w7h8x\") pod \"postgres-7cdbf6dc44-qj74q\" (UID: \"d0f23d37-4482-4829-8b98-93819e55af01\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.262843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.262802 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:12.542426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:12.542401 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q"] Oct 31 23:05:12.544766 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:05:12.544734 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f23d37_4482_4829_8b98_93819e55af01.slice/crio-164dbc40c383051b5d7952d3a256cc6496c5d441a3e33258efa876be51c122f9 WatchSource:0}: Error finding container 164dbc40c383051b5d7952d3a256cc6496c5d441a3e33258efa876be51c122f9: Status 404 returned error can't find the container with id 164dbc40c383051b5d7952d3a256cc6496c5d441a3e33258efa876be51c122f9 Oct 31 23:05:13.287064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:13.287031 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" event={"ID":"d0f23d37-4482-4829-8b98-93819e55af01","Type":"ContainerStarted","Data":"164dbc40c383051b5d7952d3a256cc6496c5d441a3e33258efa876be51c122f9"} Oct 31 23:05:14.892977 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:05:14.892940 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:05:18.315015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:18.314958 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" event={"ID":"d0f23d37-4482-4829-8b98-93819e55af01","Type":"ContainerStarted","Data":"37f54d07e063b653c87de4a56548ff00172a70d20d667c1354c015c412ab3604"} Oct 31 23:05:18.315568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:18.315108 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:18.334247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:18.334194 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" podStartSLOduration=2.024390334 podStartE2EDuration="7.33418132s" totalImagesPullingTime="5.309790986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:05:18.333853487 +0000 UTC m=+896.316595865" watchObservedRunningTime="2026-10-31 23:05:18.33418132 +0000 UTC m=+896.316923695" Oct 31 23:05:22.245983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:22.245957 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:05:22.246620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:22.246600 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:05:24.349553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:24.349526 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qj74q" Oct 31 23:05:31.703130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.703098 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:31.707933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.707914 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.717410 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.717388 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 23:05:31.718560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.718516 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vxzrb\"" Oct 31 23:05:31.718560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.718517 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 23:05:31.718712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.718607 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 23:05:31.718900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.718881 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:31.871590 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.871557 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmj2q\" (UniqueName: \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.871762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.871607 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.871813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.871757 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.972758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.972648 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmj2q\" (UniqueName: \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.972758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.972723 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.972981 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.972797 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.975201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.975176 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.975304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.975213 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:31.985014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:31.984964 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmj2q\" (UniqueName: \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\") pod \"maas-controller-579cf7c6d-jhb7d\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:32.021454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.021424 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:32.042166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.042134 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:05:32.046965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.046941 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.057148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.057087 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:05:32.174926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.174889 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ltj\" (UniqueName: \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.175122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.174938 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.175122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.175077 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.276035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.275972 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.276218 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.276084 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ltj\" (UniqueName: \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.276218 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.276114 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.278583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.278555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.278714 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.278553 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.289115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.289091 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ltj\" (UniqueName: \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\") pod \"maas-controller-69b96f8484-7gg8s\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.302370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.302346 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:32.305039 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:05:32.305014 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7821c83_d724_48ee_ac3d_e445258e95f2.slice/crio-eacf232e6c16884f1a76f69952d5e29761565e4c328dcd62087e268331745950 WatchSource:0}: Error finding container eacf232e6c16884f1a76f69952d5e29761565e4c328dcd62087e268331745950: Status 404 returned error can't find the container with id eacf232e6c16884f1a76f69952d5e29761565e4c328dcd62087e268331745950 Oct 31 23:05:32.362662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.362635 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:32.379499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.379464 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" event={"ID":"b7821c83-d724-48ee-ac3d-e445258e95f2","Type":"ContainerStarted","Data":"eacf232e6c16884f1a76f69952d5e29761565e4c328dcd62087e268331745950"} Oct 31 23:05:32.640563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:32.640535 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:05:32.642873 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:05:32.642836 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a35fd31_2d43_452e_9871_0f4d6b569aeb.slice/crio-5b8a953b4ba9c10510d800ce103ae412ee1774b7fd99720cf77455a921dc40ed WatchSource:0}: Error finding container 5b8a953b4ba9c10510d800ce103ae412ee1774b7fd99720cf77455a921dc40ed: Status 404 returned error can't find the container with id 5b8a953b4ba9c10510d800ce103ae412ee1774b7fd99720cf77455a921dc40ed Oct 31 23:05:33.387695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:33.387655 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-7gg8s" event={"ID":"0a35fd31-2d43-452e-9871-0f4d6b569aeb","Type":"ContainerStarted","Data":"5b8a953b4ba9c10510d800ce103ae412ee1774b7fd99720cf77455a921dc40ed"} Oct 31 23:05:35.401189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:35.401090 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" event={"ID":"b7821c83-d724-48ee-ac3d-e445258e95f2","Type":"ContainerStarted","Data":"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8"} Oct 31 23:05:35.401189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:35.401168 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:35.423186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:35.423125 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" podStartSLOduration=1.58826775 podStartE2EDuration="4.423108403s" totalImagesPullingTime="2.834840653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:05:35.42143149 +0000 UTC m=+913.404173869" watchObservedRunningTime="2026-10-31 23:05:35.423108403 +0000 UTC m=+913.405850773" Oct 31 23:05:36.407735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.407690 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-7gg8s" event={"ID":"0a35fd31-2d43-452e-9871-0f4d6b569aeb","Type":"ContainerStarted","Data":"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52"} Oct 31 23:05:36.408226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.407821 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:36.430037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.429957 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69b96f8484-7gg8s" podStartSLOduration=1.5934714030000001 podStartE2EDuration="4.429939296s" totalImagesPullingTime="2.836467893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:05:36.429608678 +0000 UTC m=+914.412351064" watchObservedRunningTime="2026-10-31 23:05:36.429939296 +0000 UTC m=+914.412681674" Oct 31 23:05:36.869525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.869486 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4"] Oct 31 23:05:36.873824 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.873803 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:36.882057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.881984 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 23:05:36.882193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.882138 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 23:05:36.882193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.882175 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 23:05:36.883532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.883509 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nm5qq\"" Oct 31 23:05:36.884652 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.884627 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4"] Oct 31 23:05:36.917438 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.917404 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:36.917585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.917443 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hk4v\" (UniqueName: \"kubernetes.io/projected/b829b94d-8617-461c-9338-114f98be38b9-kube-api-access-8hk4v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:36.917585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.917535 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:36.917675 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:36.917614 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b829b94d-8617-461c-9338-114f98be38b9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.018607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.018570 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.018607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.018609 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hk4v\" (UniqueName: \"kubernetes.io/projected/b829b94d-8617-461c-9338-114f98be38b9-kube-api-access-8hk4v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.018855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.018651 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.018855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.018714 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b829b94d-8617-461c-9338-114f98be38b9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.019315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.019290 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.019420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.019401 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b829b94d-8617-461c-9338-114f98be38b9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.021096 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.021077 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b829b94d-8617-461c-9338-114f98be38b9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.031386 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.031351 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hk4v\" (UniqueName: \"kubernetes.io/projected/b829b94d-8617-461c-9338-114f98be38b9-kube-api-access-8hk4v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qw4q4\" (UID: \"b829b94d-8617-461c-9338-114f98be38b9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.190182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.190090 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:37.463944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:37.463772 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4"] Oct 31 23:05:37.466698 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:05:37.466667 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb829b94d_8617_461c_9338_114f98be38b9.slice/crio-1b9179eabfd1d8d2e4e94cdad1cfab6990b85a94b2dcc61eadb41ee04227702c WatchSource:0}: Error finding container 1b9179eabfd1d8d2e4e94cdad1cfab6990b85a94b2dcc61eadb41ee04227702c: Status 404 returned error can't find the container with id 1b9179eabfd1d8d2e4e94cdad1cfab6990b85a94b2dcc61eadb41ee04227702c Oct 31 23:05:38.417303 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:38.417266 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" event={"ID":"b829b94d-8617-461c-9338-114f98be38b9","Type":"ContainerStarted","Data":"1b9179eabfd1d8d2e4e94cdad1cfab6990b85a94b2dcc61eadb41ee04227702c"} Oct 31 23:05:44.890283 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:05:44.890241 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:05:46.410582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:46.410493 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:47.414636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.414599 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:05:47.474200 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.474057 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:47.474746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.474537 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" podUID="b7821c83-d724-48ee-ac3d-e445258e95f2" containerName="manager" containerID="cri-o://4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8" gracePeriod=30 Oct 31 23:05:47.755589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.755558 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:47.822289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.822254 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\" (UniqueName: \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\") pod \"b7821c83-d724-48ee-ac3d-e445258e95f2\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " Oct 31 23:05:47.822481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.822375 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\") pod \"b7821c83-d724-48ee-ac3d-e445258e95f2\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " Oct 31 23:05:47.822481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.822457 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\") pod \"b7821c83-d724-48ee-ac3d-e445258e95f2\" (UID: \"b7821c83-d724-48ee-ac3d-e445258e95f2\") " Oct 31 23:05:47.824970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.824937 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q" pod "b7821c83-d724-48ee-ac3d-e445258e95f2" (UID: "b7821c83-d724-48ee-ac3d-e445258e95f2"). InnerVolumeSpecName "kube-api-access-pmj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:05:47.825105 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.824945 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert" pod "b7821c83-d724-48ee-ac3d-e445258e95f2" (UID: "b7821c83-d724-48ee-ac3d-e445258e95f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:05:47.825105 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.825043 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs" pod "b7821c83-d724-48ee-ac3d-e445258e95f2" (UID: "b7821c83-d724-48ee-ac3d-e445258e95f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:05:47.923457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.923418 7074 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-webhook-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:47.923457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.923455 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7821c83-d724-48ee-ac3d-e445258e95f2-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:47.923685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:47.923466 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmj2q\" (UniqueName: \"kubernetes.io/projected/b7821c83-d724-48ee-ac3d-e445258e95f2-kube-api-access-pmj2q\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:05:48.501707 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.501676 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" event={"ID":"b829b94d-8617-461c-9338-114f98be38b9","Type":"ContainerStarted","Data":"f9212cc027c2b6949f1fd895ab52fd5e003d0abb169ebd16a91153fff70e85fb"} Oct 31 23:05:48.502201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.501774 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:05:48.503080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.503058 7074 generic.go:356] "Generic (PLEG): container finished" podID="b7821c83-d724-48ee-ac3d-e445258e95f2" containerID="4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8" exitCode=0 Oct 31 23:05:48.503170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.503099 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" Oct 31 23:05:48.503170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.503147 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" event={"ID":"b7821c83-d724-48ee-ac3d-e445258e95f2","Type":"ContainerDied","Data":"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8"} Oct 31 23:05:48.503243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.503195 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jhb7d" event={"ID":"b7821c83-d724-48ee-ac3d-e445258e95f2","Type":"ContainerDied","Data":"eacf232e6c16884f1a76f69952d5e29761565e4c328dcd62087e268331745950"} Oct 31 23:05:48.503243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.503222 7074 scope.go:122] "RemoveContainer" containerID="4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8" Oct 31 23:05:48.517349 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.517328 7074 scope.go:122] "RemoveContainer" containerID="4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8" Oct 31 23:05:48.517607 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:05:48.517589 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8\": container with ID starting with 4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8 not found: ID does not exist" containerID="4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8" Oct 31 23:05:48.517702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.517616 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8"} err="failed to get container status \"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8\": rpc error: code = NotFound desc = could not find container \"4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8\": container with ID starting with 4b2aca03e910586a48552a22c853b3df81eefe70096568fcfab047cbed8d0ef8 not found: ID does not exist" Oct 31 23:05:48.523308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.523271 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" podStartSLOduration=1.645824573 podStartE2EDuration="12.523259234s" totalImagesPullingTime="10.877434661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:05:48.521133859 +0000 UTC m=+926.503876237" watchObservedRunningTime="2026-10-31 23:05:48.523259234 +0000 UTC m=+926.506001611" Oct 31 23:05:48.536641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.536622 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:48.540772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:48.540752 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jhb7d"] Oct 31 23:05:50.140520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:50.140484 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7821c83-d724-48ee-ac3d-e445258e95f2" path="/var/lib/kubelet/pods/b7821c83-d724-48ee-ac3d-e445258e95f2/volumes" Oct 31 23:05:59.514647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:05:59.514610 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" Oct 31 23:06:08.771163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.771129 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:08.771698 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.771591 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b7821c83-d724-48ee-ac3d-e445258e95f2" containerName="manager" Oct 31 23:06:08.771698 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.771605 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7821c83-d724-48ee-ac3d-e445258e95f2" containerName="manager" Oct 31 23:06:08.771698 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.771681 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7821c83-d724-48ee-ac3d-e445258e95f2" containerName="manager" Oct 31 23:06:08.774613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.774596 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:08.781142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.781121 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 23:06:08.781259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.781160 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\"" Oct 31 23:06:08.781259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.781181 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 23:06:08.786719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.786696 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:08.905671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.905639 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:08.905846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.905689 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:08.905846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.905788 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:08.905918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:08.905896 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgl85\" (UniqueName: \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.006935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.006899 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.007134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.006950 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.007134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.007028 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgl85\" (UniqueName: \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.007134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.007058 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.007620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.007589 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.009490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.009468 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.009596 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.009500 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.019619 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.019600 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgl85\" (UniqueName: \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\") pod \"maas-api-6d669c7fcd-xpknb\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.087900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.087833 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:09.362789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.362649 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:09.366161 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:06:09.366137 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac75044_9aae_4a4e_885e_b507995c58b0.slice/crio-bb7a45887fe4b22c31a1e7f8ec8be0fa358335bc6c261d1f43335fd9cd56ac01 WatchSource:0}: Error finding container bb7a45887fe4b22c31a1e7f8ec8be0fa358335bc6c261d1f43335fd9cd56ac01: Status 404 returned error can't find the container with id bb7a45887fe4b22c31a1e7f8ec8be0fa358335bc6c261d1f43335fd9cd56ac01 Oct 31 23:06:09.603538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:09.603486 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" event={"ID":"3ac75044-9aae-4a4e-885e-b507995c58b0","Type":"ContainerStarted","Data":"bb7a45887fe4b22c31a1e7f8ec8be0fa358335bc6c261d1f43335fd9cd56ac01"} Oct 31 23:06:11.618338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:11.618300 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" event={"ID":"3ac75044-9aae-4a4e-885e-b507995c58b0","Type":"ContainerStarted","Data":"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7"} Oct 31 23:06:11.618807 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:11.618366 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:14.887108 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:14.887071 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:06:17.629348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:17.629317 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:17.650556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:17.650504 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" podStartSLOduration=8.199189403 podStartE2EDuration="9.650490581s" totalImagesPullingTime="1.451301178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:06:11.658683665 +0000 UTC m=+949.641426041" watchObservedRunningTime="2026-10-31 23:06:17.650490581 +0000 UTC m=+955.633232937" Oct 31 23:06:19.351077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.351050 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-scm8s"] Oct 31 23:06:19.513663 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.513626 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-scm8s"] Oct 31 23:06:19.513663 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.513669 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-scm8s"] Oct 31 23:06:19.513919 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.513755 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.514216 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:19.514192 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mfk2b tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-mfk2b tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-scm8s" podUID="a93a8b09-4357-4908-9847-a5447e2e7f77" Oct 31 23:06:19.609670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.609593 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.609848 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.609664 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfk2b\" (UniqueName: \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.662494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.662462 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.697484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.697445 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.710502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.710462 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.710662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.710541 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfk2b\" (UniqueName: \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.710662 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:19.710638 7074 secret.go:189] Couldn't get secret kuadrant-system/authorino-server-cert: object "kuadrant-system"/"authorino-server-cert" not registered Oct 31 23:06:19.710730 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:19.710717 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert podName:a93a8b09-4357-4908-9847-a5447e2e7f77 nodeName:}" failed. No retries permitted until 2026-10-31 23:06:20.210698662 +0000 UTC m=+958.193441037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert") pod "authorino-6dd76875b5-scm8s" (UID: "a93a8b09-4357-4908-9847-a5447e2e7f77") : object "kuadrant-system"/"authorino-server-cert" not registered Oct 31 23:06:19.727376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.727325 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfk2b\" (UniqueName: \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:19.786138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.786106 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:06:19.816621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.816585 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:06:19.816804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.816684 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:19.820821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.820791 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p44kh\"" Oct 31 23:06:19.820821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.820817 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 23:06:19.913101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.912984 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\" (UniqueName: \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\") pod \"a93a8b09-4357-4908-9847-a5447e2e7f77\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " Oct 31 23:06:19.913261 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.913165 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:19.913261 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.913236 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54tq\" (UniqueName: \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:19.915132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:19.915104 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b" pod "a93a8b09-4357-4908-9847-a5447e2e7f77" (UID: "a93a8b09-4357-4908-9847-a5447e2e7f77"). InnerVolumeSpecName "kube-api-access-mfk2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:06:20.014079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.014038 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:20.014251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.014124 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54tq\" (UniqueName: \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:20.014251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.014161 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfk2b\" (UniqueName: \"kubernetes.io/projected/a93a8b09-4357-4908-9847-a5447e2e7f77-kube-api-access-mfk2b\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:20.016667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.016635 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:20.044806 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.044769 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54tq\" (UniqueName: \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\") pod \"authorino-67fb758fbc-2ztvk\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:20.147083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.147052 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:06:20.216601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.216562 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:20.219688 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.219635 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") pod \"authorino-6dd76875b5-scm8s\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:20.317938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.317908 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") pod \"a93a8b09-4357-4908-9847-a5447e2e7f77\" (UID: \"a93a8b09-4357-4908-9847-a5447e2e7f77\") " Oct 31 23:06:20.320018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.319980 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert" pod "a93a8b09-4357-4908-9847-a5447e2e7f77" (UID: "a93a8b09-4357-4908-9847-a5447e2e7f77"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:06:20.419359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.419295 7074 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a93a8b09-4357-4908-9847-a5447e2e7f77-tls-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:20.431566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.431540 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:06:20.434247 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:06:20.434221 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0efe9c9_0fc9_427e_ab0b_2cdc08a4e071.slice/crio-405de0722e920c42b6fedc46b52231efa849aa5cbd870a3176c5d63f0a109ffb WatchSource:0}: Error finding container 405de0722e920c42b6fedc46b52231efa849aa5cbd870a3176c5d63f0a109ffb: Status 404 returned error can't find the container with id 405de0722e920c42b6fedc46b52231efa849aa5cbd870a3176c5d63f0a109ffb Oct 31 23:06:20.668543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.668517 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-scm8s" Oct 31 23:06:20.668711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.668547 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" event={"ID":"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071","Type":"ContainerStarted","Data":"405de0722e920c42b6fedc46b52231efa849aa5cbd870a3176c5d63f0a109ffb"} Oct 31 23:06:20.697351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.697275 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-scm8s"] Oct 31 23:06:20.703082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:20.703059 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-scm8s"] Oct 31 23:06:21.676104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:21.676058 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" event={"ID":"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071","Type":"ContainerStarted","Data":"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92"} Oct 31 23:06:21.697224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:21.697160 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" podStartSLOduration=1.914764275 podStartE2EDuration="2.697141387s" totalImagesPullingTime="782.377112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:06:21.69512702 +0000 UTC m=+959.677869397" watchObservedRunningTime="2026-10-31 23:06:21.697141387 +0000 UTC m=+959.679883765" Oct 31 23:06:22.140957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:22.140927 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93a8b09-4357-4908-9847-a5447e2e7f77" path="/var/lib/kubelet/pods/a93a8b09-4357-4908-9847-a5447e2e7f77/volumes" Oct 31 23:06:29.311867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.311831 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:29.312353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.312099 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" podUID="3ac75044-9aae-4a4e-885e-b507995c58b0" containerName="maas-api" containerID="cri-o://1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7" gracePeriod=30 Oct 31 23:06:29.555409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.555386 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:29.598595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.598517 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\") pod \"3ac75044-9aae-4a4e-885e-b507995c58b0\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " Oct 31 23:06:29.598751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.598677 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\") pod \"3ac75044-9aae-4a4e-885e-b507995c58b0\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " Oct 31 23:06:29.598751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.598710 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\") pod \"3ac75044-9aae-4a4e-885e-b507995c58b0\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " Oct 31 23:06:29.598873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.598839 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\" (UniqueName: \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\") pod \"3ac75044-9aae-4a4e-885e-b507995c58b0\" (UID: \"3ac75044-9aae-4a4e-885e-b507995c58b0\") " Oct 31 23:06:29.599089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.599052 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca" pod "3ac75044-9aae-4a4e-885e-b507995c58b0" (UID: "3ac75044-9aae-4a4e-885e-b507995c58b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:06:29.599200 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.599187 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75044-9aae-4a4e-885e-b507995c58b0-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:29.600973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.600934 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs" pod "3ac75044-9aae-4a4e-885e-b507995c58b0" (UID: "3ac75044-9aae-4a4e-885e-b507995c58b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:06:29.600973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.600948 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls" pod "3ac75044-9aae-4a4e-885e-b507995c58b0" (UID: "3ac75044-9aae-4a4e-885e-b507995c58b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:06:29.601447 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.601423 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85" pod "3ac75044-9aae-4a4e-885e-b507995c58b0" (UID: "3ac75044-9aae-4a4e-885e-b507995c58b0"). InnerVolumeSpecName "kube-api-access-dgl85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:06:29.700454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.700418 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:29.700454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.700445 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgl85\" (UniqueName: \"kubernetes.io/projected/3ac75044-9aae-4a4e-885e-b507995c58b0-kube-api-access-dgl85\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:29.700454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.700457 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac75044-9aae-4a4e-885e-b507995c58b0-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:29.727385 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.727354 7074 generic.go:356] "Generic (PLEG): container finished" podID="3ac75044-9aae-4a4e-885e-b507995c58b0" containerID="1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7" exitCode=0 Oct 31 23:06:29.727560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.727396 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" Oct 31 23:06:29.727560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.727442 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" event={"ID":"3ac75044-9aae-4a4e-885e-b507995c58b0","Type":"ContainerDied","Data":"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7"} Oct 31 23:06:29.727560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.727502 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb" event={"ID":"3ac75044-9aae-4a4e-885e-b507995c58b0","Type":"ContainerDied","Data":"bb7a45887fe4b22c31a1e7f8ec8be0fa358335bc6c261d1f43335fd9cd56ac01"} Oct 31 23:06:29.727560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.727546 7074 scope.go:122] "RemoveContainer" containerID="1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7" Oct 31 23:06:29.738821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.738803 7074 scope.go:122] "RemoveContainer" containerID="1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7" Oct 31 23:06:29.739335 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:29.739316 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7\": container with ID starting with 1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7 not found: ID does not exist" containerID="1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7" Oct 31 23:06:29.739434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.739345 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7"} err="failed to get container status \"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7\": rpc error: code = NotFound desc = could not find container \"1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7\": container with ID starting with 1b2992172cf840ffc8a525f6aeb4fc48d1946a2122158e098ba7e9ba15d557b7 not found: ID does not exist" Oct 31 23:06:29.752913 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.752893 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:29.756915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:29.756895 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xpknb"] Oct 31 23:06:30.140771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:06:30.140713 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac75044-9aae-4a4e-885e-b507995c58b0" path="/var/lib/kubelet/pods/3ac75044-9aae-4a4e-885e-b507995c58b0/volumes" Oct 31 23:06:44.887743 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:06:44.887703 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:07:12.843631 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.843545 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6"] Oct 31 23:07:12.844236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.844215 7074 cpu_manager.go:414] "containerMap: removing container" podUID="3ac75044-9aae-4a4e-885e-b507995c58b0" containerName="maas-api" Oct 31 23:07:12.844320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.844238 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac75044-9aae-4a4e-885e-b507995c58b0" containerName="maas-api" Oct 31 23:07:12.844374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.844339 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac75044-9aae-4a4e-885e-b507995c58b0" containerName="maas-api" Oct 31 23:07:12.847779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.847753 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.853277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.853252 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 23:07:12.853449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.853416 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 23:07:12.855270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.855252 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ltzbh\"" Oct 31 23:07:12.855364 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.855252 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 23:07:12.862733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.862703 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6"] Oct 31 23:07:12.888632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.888590 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.888852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.888647 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.888852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.888771 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.888983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.888930 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1586d3a6-8548-4363-9e8c-0b5878e019f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.889069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.889054 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqf24\" (UniqueName: \"kubernetes.io/projected/1586d3a6-8548-4363-9e8c-0b5878e019f0-kube-api-access-vqf24\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.889113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.889092 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.989967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.989932 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqf24\" (UniqueName: \"kubernetes.io/projected/1586d3a6-8548-4363-9e8c-0b5878e019f0-kube-api-access-vqf24\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.989967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.989973 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990018 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990037 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990054 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990084 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1586d3a6-8548-4363-9e8c-0b5878e019f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990368 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990447 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.990489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.990478 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.992409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.992385 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1586d3a6-8548-4363-9e8c-0b5878e019f0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:12.992713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:12.992695 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1586d3a6-8548-4363-9e8c-0b5878e019f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:13.002515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:13.002487 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqf24\" (UniqueName: \"kubernetes.io/projected/1586d3a6-8548-4363-9e8c-0b5878e019f0-kube-api-access-vqf24\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lqrb6\" (UID: \"1586d3a6-8548-4363-9e8c-0b5878e019f0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:13.163040 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:13.162926 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:13.451351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:13.451273 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6"] Oct 31 23:07:13.454771 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:13.454738 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1586d3a6_8548_4363_9e8c_0b5878e019f0.slice/crio-9f0bfc77012be48c6eb4095bf267e885ea3d5fb983482dc469851efe135edcac WatchSource:0}: Error finding container 9f0bfc77012be48c6eb4095bf267e885ea3d5fb983482dc469851efe135edcac: Status 404 returned error can't find the container with id 9f0bfc77012be48c6eb4095bf267e885ea3d5fb983482dc469851efe135edcac Oct 31 23:07:13.946614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:13.946563 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" event={"ID":"1586d3a6-8548-4363-9e8c-0b5878e019f0","Type":"ContainerStarted","Data":"9f0bfc77012be48c6eb4095bf267e885ea3d5fb983482dc469851efe135edcac"} Oct 31 23:07:14.888470 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:07:14.888430 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:07:14.955569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:14.955517 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" event={"ID":"1586d3a6-8548-4363-9e8c-0b5878e019f0","Type":"ContainerStarted","Data":"fde4ff8acc527cbdcefcd5902c615fd53fbd84b74a43c33189bb9e6f9159f695"} Oct 31 23:07:14.977493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:14.977437 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" podStartSLOduration=1.602395491 podStartE2EDuration="2.97742257s" totalImagesPullingTime="1.375027079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:14.976270763 +0000 UTC m=+1012.959013141" watchObservedRunningTime="2026-10-31 23:07:14.97742257 +0000 UTC m=+1012.960164943" Oct 31 23:07:15.163789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:15.163697 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:18.174960 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.174926 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg"] Oct 31 23:07:18.178695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.178675 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.190344 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.190323 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 23:07:18.194704 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.194677 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg"] Oct 31 23:07:18.244185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244154 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.244353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244191 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.244353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244214 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa2b701-0d76-4fb1-9246-d86cd4589868-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.244424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244350 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.244424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244402 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.244498 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.244477 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vb26\" (UniqueName: \"kubernetes.io/projected/8aa2b701-0d76-4fb1-9246-d86cd4589868-kube-api-access-9vb26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345214 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345254 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vb26\" (UniqueName: \"kubernetes.io/projected/8aa2b701-0d76-4fb1-9246-d86cd4589868-kube-api-access-9vb26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345325 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345344 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345364 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa2b701-0d76-4fb1-9246-d86cd4589868-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345393 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345858 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345775 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.345858 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345816 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.346076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.345887 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.347703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.347683 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa2b701-0d76-4fb1-9246-d86cd4589868-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.347887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.347868 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa2b701-0d76-4fb1-9246-d86cd4589868-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.364865 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.364830 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vb26\" (UniqueName: \"kubernetes.io/projected/8aa2b701-0d76-4fb1-9246-d86cd4589868-kube-api-access-9vb26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg\" (UID: \"8aa2b701-0d76-4fb1-9246-d86cd4589868\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.492877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.492791 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:18.770814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.770750 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg"] Oct 31 23:07:18.773115 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:18.773090 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2b701_0d76_4fb1_9246_d86cd4589868.slice/crio-638b9e63b16465575e66a121d129ae450b79393ad23f47b133e790a4458007c6 WatchSource:0}: Error finding container 638b9e63b16465575e66a121d129ae450b79393ad23f47b133e790a4458007c6: Status 404 returned error can't find the container with id 638b9e63b16465575e66a121d129ae450b79393ad23f47b133e790a4458007c6 Oct 31 23:07:18.975775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:18.975719 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" event={"ID":"8aa2b701-0d76-4fb1-9246-d86cd4589868","Type":"ContainerStarted","Data":"638b9e63b16465575e66a121d129ae450b79393ad23f47b133e790a4458007c6"} Oct 31 23:07:19.981890 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:19.981853 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" event={"ID":"8aa2b701-0d76-4fb1-9246-d86cd4589868","Type":"ContainerStarted","Data":"219efcb44058956b369e5912e914b969617376ada8c1906a0905706023bf6cb6"} Oct 31 23:07:20.005091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:20.005034 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" podStartSLOduration=1.725958709 podStartE2EDuration="2.005017663s" totalImagesPullingTime="279.058954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:20.003008647 +0000 UTC m=+1017.985751020" watchObservedRunningTime="2026-10-31 23:07:20.005017663 +0000 UTC m=+1017.987760037" Oct 31 23:07:20.492914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:20.492882 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:22.260313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.260283 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr"] Oct 31 23:07:22.265275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.265253 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.285178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.285153 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 23:07:22.302739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.302712 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr"] Oct 31 23:07:22.382735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.382698 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.382904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.382814 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.382904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.382864 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.383047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.383016 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmb62\" (UniqueName: \"kubernetes.io/projected/937794d1-b7e8-4afe-8915-c4916cd7c9cf-kube-api-access-dmb62\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.383090 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.383051 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.383134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.383111 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.483978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.483942 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmb62\" (UniqueName: \"kubernetes.io/projected/937794d1-b7e8-4afe-8915-c4916cd7c9cf-kube-api-access-dmb62\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.483978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.483983 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484037 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484085 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484141 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484198 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484532 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484535 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.484621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.484590 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.486437 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.486410 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937794d1-b7e8-4afe-8915-c4916cd7c9cf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.486574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.486557 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937794d1-b7e8-4afe-8915-c4916cd7c9cf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.499064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.499035 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmb62\" (UniqueName: \"kubernetes.io/projected/937794d1-b7e8-4afe-8915-c4916cd7c9cf-kube-api-access-dmb62\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr\" (UID: \"937794d1-b7e8-4afe-8915-c4916cd7c9cf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.577185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.577107 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:22.850675 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.850648 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr"] Oct 31 23:07:22.853144 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:22.853119 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937794d1_b7e8_4afe_8915_c4916cd7c9cf.slice/crio-504ae9d39776a5f4ebfa8d539680389cf87fb5d99a3ae33e7a6ef31270e1b133 WatchSource:0}: Error finding container 504ae9d39776a5f4ebfa8d539680389cf87fb5d99a3ae33e7a6ef31270e1b133: Status 404 returned error can't find the container with id 504ae9d39776a5f4ebfa8d539680389cf87fb5d99a3ae33e7a6ef31270e1b133 Oct 31 23:07:22.998126 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:22.998094 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" event={"ID":"937794d1-b7e8-4afe-8915-c4916cd7c9cf","Type":"ContainerStarted","Data":"504ae9d39776a5f4ebfa8d539680389cf87fb5d99a3ae33e7a6ef31270e1b133"} Oct 31 23:07:23.163881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.163845 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:23.177250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.177226 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:23.300299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.300262 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr"] Oct 31 23:07:23.304551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.304518 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.314082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.313799 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 23:07:23.318594 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.318564 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr"] Oct 31 23:07:23.393136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393042 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7vf\" (UniqueName: \"kubernetes.io/projected/de3707bc-9fcd-4ae0-8942-1b95d920d785-kube-api-access-vt7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.393136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393086 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.393331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393190 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.393331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393283 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.393331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393311 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3707bc-9fcd-4ae0-8942-1b95d920d785-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.393486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.393441 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500638 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500696 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3707bc-9fcd-4ae0-8942-1b95d920d785-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500785 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500860 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7vf\" (UniqueName: \"kubernetes.io/projected/de3707bc-9fcd-4ae0-8942-1b95d920d785-kube-api-access-vt7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500889 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.501009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.500940 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.508012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.503562 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.508012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.503861 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.508012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.504071 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.527009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.520265 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3707bc-9fcd-4ae0-8942-1b95d920d785-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.527009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.526626 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3707bc-9fcd-4ae0-8942-1b95d920d785-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.531007 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.528486 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7vf\" (UniqueName: \"kubernetes.io/projected/de3707bc-9fcd-4ae0-8942-1b95d920d785-kube-api-access-vt7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr\" (UID: \"de3707bc-9fcd-4ae0-8942-1b95d920d785\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.621969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.621934 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:23.908536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:23.908397 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr"] Oct 31 23:07:23.911141 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:23.911113 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3707bc_9fcd_4ae0_8942_1b95d920d785.slice/crio-7ca3f0888bba33901cdf37ea3b4d44371b5709d26b3a4486463bd5042be20e79 WatchSource:0}: Error finding container 7ca3f0888bba33901cdf37ea3b4d44371b5709d26b3a4486463bd5042be20e79: Status 404 returned error can't find the container with id 7ca3f0888bba33901cdf37ea3b4d44371b5709d26b3a4486463bd5042be20e79 Oct 31 23:07:24.006396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:24.006364 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" event={"ID":"937794d1-b7e8-4afe-8915-c4916cd7c9cf","Type":"ContainerStarted","Data":"104d5bab46b0dd1fc29d710948e9b60314034d16ff5d2ef006de74a3b75a2681"} Oct 31 23:07:24.007691 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:24.007666 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" event={"ID":"de3707bc-9fcd-4ae0-8942-1b95d920d785","Type":"ContainerStarted","Data":"7ca3f0888bba33901cdf37ea3b4d44371b5709d26b3a4486463bd5042be20e79"} Oct 31 23:07:24.019239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:24.019221 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lqrb6" Oct 31 23:07:24.030411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:24.030371 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podStartSLOduration=1.748008775 podStartE2EDuration="2.030360657s" totalImagesPullingTime="282.351882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:24.029306411 +0000 UTC m=+1022.012048789" watchObservedRunningTime="2026-10-31 23:07:24.030360657 +0000 UTC m=+1022.013103033" Oct 31 23:07:24.578100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:24.578067 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:25.016695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:25.016655 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" event={"ID":"de3707bc-9fcd-4ae0-8942-1b95d920d785","Type":"ContainerStarted","Data":"d2a0c2c4d887e54aa8a4d4c8b2563c664e450e8905f958b092a33ca63f92fbef"} Oct 31 23:07:25.039007 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:25.038944 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" podStartSLOduration=1.7487004210000001 podStartE2EDuration="2.038930764s" totalImagesPullingTime="290.230343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:25.037130659 +0000 UTC m=+1023.019873073" watchObservedRunningTime="2026-10-31 23:07:25.038930764 +0000 UTC m=+1023.021673140" Oct 31 23:07:25.622415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:25.622358 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:28.493140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:28.493099 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:28.505677 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:28.505644 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:29.047076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:29.047050 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg" Oct 31 23:07:32.577394 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:32.577342 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:32.589623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:32.589594 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:33.067405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:33.067374 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:07:33.622171 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:33.622136 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:33.634485 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:33.634456 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:34.071328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:34.071301 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr" Oct 31 23:07:42.192846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.192808 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:07:42.193396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.192973 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" podUID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" containerName="authorino" containerID="cri-o://1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92" gracePeriod=30 Oct 31 23:07:42.438556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.438529 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:07:42.583235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.583209 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\" (UniqueName: \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\") pod \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " Oct 31 23:07:42.583432 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.583348 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\" (UniqueName: \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\") pod \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\" (UID: \"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071\") " Oct 31 23:07:42.585439 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.585403 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq" pod "c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" (UID: "c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071"). InnerVolumeSpecName "kube-api-access-c54tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:07:42.593774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.593750 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert" pod "c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" (UID: "c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:07:42.684883 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.684849 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54tq\" (UniqueName: \"kubernetes.io/projected/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-kube-api-access-c54tq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:07:42.684883 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.684880 7074 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071-tls-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:07:42.767724 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.767687 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:07:42.768125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.768100 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" containerName="authorino" Oct 31 23:07:42.768125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.768114 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" containerName="authorino" Oct 31 23:07:42.768294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.768225 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" containerName="authorino" Oct 31 23:07:42.772287 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.772269 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:42.776647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.776625 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 23:07:42.776739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.776674 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 23:07:42.776739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.776719 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-pnfhd\"" Oct 31 23:07:42.782174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.782153 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:07:42.887312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.887218 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:42.887312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.887287 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526tx\" (UniqueName: \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:42.987913 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.987880 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526tx\" (UniqueName: \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:42.988082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.987983 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:42.990377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:42.990358 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:43.005163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.005136 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526tx\" (UniqueName: \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\") pod \"ai-gateway-controller-7d6c696b49-28vck\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:43.087924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.087895 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:43.108054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.108026 7074 generic.go:356] "Generic (PLEG): container finished" podID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" containerID="1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92" exitCode=0 Oct 31 23:07:43.108182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.108070 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" Oct 31 23:07:43.108182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.108113 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" event={"ID":"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071","Type":"ContainerDied","Data":"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92"} Oct 31 23:07:43.108182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.108161 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fb758fbc-2ztvk" event={"ID":"c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071","Type":"ContainerDied","Data":"405de0722e920c42b6fedc46b52231efa849aa5cbd870a3176c5d63f0a109ffb"} Oct 31 23:07:43.108348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.108183 7074 scope.go:122] "RemoveContainer" containerID="1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92" Oct 31 23:07:43.120371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.120351 7074 scope.go:122] "RemoveContainer" containerID="1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92" Oct 31 23:07:43.121639 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:07:43.121614 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92\": container with ID starting with 1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92 not found: ID does not exist" containerID="1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92" Oct 31 23:07:43.121721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.121651 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92"} err="failed to get container status \"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92\": rpc error: code = NotFound desc = could not find container \"1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92\": container with ID starting with 1200d6409eff2f041fcdcb253c8332dc8a7f8c6d51a6a1fee2da83a89bef3f92 not found: ID does not exist" Oct 31 23:07:43.135928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.135902 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:07:43.146655 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.146630 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-67fb758fbc-2ztvk"] Oct 31 23:07:43.374260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:43.374233 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:07:43.377230 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:43.377198 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf847ea5e_71be_435b_b177_775b4ebd9850.slice/crio-b48bff13b9a22b7ea2f5ef4b9626f611dcd90555cb9f492e0df1f6e474e8e082 WatchSource:0}: Error finding container b48bff13b9a22b7ea2f5ef4b9626f611dcd90555cb9f492e0df1f6e474e8e082: Status 404 returned error can't find the container with id b48bff13b9a22b7ea2f5ef4b9626f611dcd90555cb9f492e0df1f6e474e8e082 Oct 31 23:07:44.116909 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:44.116867 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" event={"ID":"f847ea5e-71be-435b-b177-775b4ebd9850","Type":"ContainerStarted","Data":"b48bff13b9a22b7ea2f5ef4b9626f611dcd90555cb9f492e0df1f6e474e8e082"} Oct 31 23:07:44.142322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:44.142289 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071" path="/var/lib/kubelet/pods/c0efe9c9-0fc9-427e-ab0b-2cdc08a4e071/volumes" Oct 31 23:07:44.887046 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:07:44.887013 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:07:45.124393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:45.124303 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" event={"ID":"f847ea5e-71be-435b-b177-775b4ebd9850","Type":"ContainerStarted","Data":"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd"} Oct 31 23:07:45.124545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:45.124416 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:45.145288 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:45.145240 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" podStartSLOduration=1.748269379 podStartE2EDuration="3.145226677s" totalImagesPullingTime="1.396957298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:45.143950862 +0000 UTC m=+1043.126693239" watchObservedRunningTime="2026-10-31 23:07:45.145226677 +0000 UTC m=+1043.127969084" Oct 31 23:07:46.560726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.560692 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp"] Oct 31 23:07:46.564721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.564701 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.574262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.574236 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 23:07:46.574556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.574533 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-66c2d\"" Oct 31 23:07:46.576780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.576763 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp"] Oct 31 23:07:46.603817 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.603786 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7ql7n"] Oct 31 23:07:46.607531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.607512 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.617947 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.617922 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7ql7n"] Oct 31 23:07:46.725588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.725552 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5rw\" (UniqueName: \"kubernetes.io/projected/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-kube-api-access-7b5rw\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.725766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.725615 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-plugins-config-volume\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.725815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.725757 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b58f6ad-0951-4522-bb08-2c57d176c2b3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.725815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.725795 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/8b58f6ad-0951-4522-bb08-2c57d176c2b3-kube-api-access-mfr4t\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.826803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.826726 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5rw\" (UniqueName: \"kubernetes.io/projected/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-kube-api-access-7b5rw\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.826969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.826799 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-plugins-config-volume\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.826969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.826896 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b58f6ad-0951-4522-bb08-2c57d176c2b3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.826969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.826942 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/8b58f6ad-0951-4522-bb08-2c57d176c2b3-kube-api-access-mfr4t\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.827678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.827651 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b58f6ad-0951-4522-bb08-2c57d176c2b3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.827678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.827663 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-plugins-config-volume\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.839102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.839078 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5rw\" (UniqueName: \"kubernetes.io/projected/656d49b3-6eff-4b88-8719-08e5fdbfd7d5-kube-api-access-7b5rw\") pod \"payload-processing-7bb9489676-7ql7n\" (UID: \"656d49b3-6eff-4b88-8719-08e5fdbfd7d5\") " pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:46.843157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.843130 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/8b58f6ad-0951-4522-bb08-2c57d176c2b3-kube-api-access-mfr4t\") pod \"payload-pre-processing-7b98bb678c-xdxrp\" (UID: \"8b58f6ad-0951-4522-bb08-2c57d176c2b3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.880043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.879982 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:46.922961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:46.922927 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:47.157634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.157606 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp"] Oct 31 23:07:47.159816 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:47.159788 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b58f6ad_0951_4522_bb08_2c57d176c2b3.slice/crio-4944dfd201db5d517dc70a70537553f3315714453efefcef70be9d1988398f6f WatchSource:0}: Error finding container 4944dfd201db5d517dc70a70537553f3315714453efefcef70be9d1988398f6f: Status 404 returned error can't find the container with id 4944dfd201db5d517dc70a70537553f3315714453efefcef70be9d1988398f6f Oct 31 23:07:47.229528 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.229504 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-7ql7n"] Oct 31 23:07:47.231420 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:47.231389 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656d49b3_6eff_4b88_8719_08e5fdbfd7d5.slice/crio-92d097a34a214b7714cc0c1962249da17b9a70fc9ac97c59a93329d39c636d21 WatchSource:0}: Error finding container 92d097a34a214b7714cc0c1962249da17b9a70fc9ac97c59a93329d39c636d21: Status 404 returned error can't find the container with id 92d097a34a214b7714cc0c1962249da17b9a70fc9ac97c59a93329d39c636d21 Oct 31 23:07:47.273387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.273349 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:07:47.279961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.279938 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.295977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.295953 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nf6c6\"" Oct 31 23:07:47.302933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.302905 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:07:47.433803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433721 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.433803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433760 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.433803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433779 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433849 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hhj\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433927 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.433954 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.434070 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.434131 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.434270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.434158 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.534954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.534913 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.534954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.534946 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.534966 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hhj\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535065 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535100 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535137 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535209 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535363 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535641 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535710 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535857 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535829 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535737 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.535911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.535761 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.536116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.536095 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.537439 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.537420 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.538060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.538044 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.547897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.547870 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hhj\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.548634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.548612 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dvbhd\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.593976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.593939 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:47.899299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.898681 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:07:47.902014 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:47.901957 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7270a3ff_1d40_4ab1_b7f6_f6178e060405.slice/crio-733ced5ffa8c90785b95b57768c361b34948932a3afebf2a4596f2d6c0f210f3 WatchSource:0}: Error finding container 733ced5ffa8c90785b95b57768c361b34948932a3afebf2a4596f2d6c0f210f3: Status 404 returned error can't find the container with id 733ced5ffa8c90785b95b57768c361b34948932a3afebf2a4596f2d6c0f210f3 Oct 31 23:07:47.904449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.904416 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:07:47.904534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.904480 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:07:47.904534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:47.904512 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:07:48.148922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.148840 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" event={"ID":"8b58f6ad-0951-4522-bb08-2c57d176c2b3","Type":"ContainerStarted","Data":"4944dfd201db5d517dc70a70537553f3315714453efefcef70be9d1988398f6f"} Oct 31 23:07:48.150291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.150263 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" event={"ID":"656d49b3-6eff-4b88-8719-08e5fdbfd7d5","Type":"ContainerStarted","Data":"92d097a34a214b7714cc0c1962249da17b9a70fc9ac97c59a93329d39c636d21"} Oct 31 23:07:48.152614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.152514 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" event={"ID":"7270a3ff-1d40-4ab1-b7f6-f6178e060405","Type":"ContainerStarted","Data":"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3"} Oct 31 23:07:48.152614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.152570 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" event={"ID":"7270a3ff-1d40-4ab1-b7f6-f6178e060405","Type":"ContainerStarted","Data":"733ced5ffa8c90785b95b57768c361b34948932a3afebf2a4596f2d6c0f210f3"} Oct 31 23:07:48.188843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.188769 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podStartSLOduration=1.188748804 podStartE2EDuration="1.188748804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:48.188615942 +0000 UTC m=+1046.171358324" watchObservedRunningTime="2026-10-31 23:07:48.188748804 +0000 UTC m=+1046.171491183" Oct 31 23:07:48.594649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.594560 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:48.596054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.596024 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:48.596187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:48.596083 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:49.594862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:49.594823 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:49.595375 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:49.594883 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:50.184422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.184388 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" event={"ID":"656d49b3-6eff-4b88-8719-08e5fdbfd7d5","Type":"ContainerStarted","Data":"6d1f4686540b41508199b984d4d99456454f2096e90c13094c7f065bb93ea1c1"} Oct 31 23:07:50.184612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.184472 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:07:50.185956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.185934 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" event={"ID":"8b58f6ad-0951-4522-bb08-2c57d176c2b3","Type":"ContainerStarted","Data":"86d7980bb4696d250b6b284c176c83f494ecfb59d08e0e798a61d9ba657079c4"} Oct 31 23:07:50.186170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.186154 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:07:50.204832 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.204752 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" podStartSLOduration=1.43545604 podStartE2EDuration="4.204737816s" totalImagesPullingTime="2.769281776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:50.203515026 +0000 UTC m=+1048.186257400" watchObservedRunningTime="2026-10-31 23:07:50.204737816 +0000 UTC m=+1048.187480190" Oct 31 23:07:50.226229 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.226185 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" podStartSLOduration=1.3067199010000001 podStartE2EDuration="4.226173894s" totalImagesPullingTime="2.919453993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:50.224920801 +0000 UTC m=+1048.207663178" watchObservedRunningTime="2026-10-31 23:07:50.226173894 +0000 UTC m=+1048.208916268" Oct 31 23:07:50.594401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.594357 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:50.594571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:50.594447 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:51.594571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:51.594536 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:51.594979 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:51.594589 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:52.594762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:52.594729 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:52.595179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:52.594782 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:53.595314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:53.595274 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:53.595818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:53.595333 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:54.594741 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:54.594706 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 23:07:54.594918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:54.594758 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 23:07:56.132426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.132389 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:07:56.313901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.313868 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:56.314939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.314920 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:56.315597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.315579 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:07:56.403004 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.402912 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d58dbf686-blgqt"] Oct 31 23:07:56.407101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.407081 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.419726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.419705 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d58dbf686-blgqt"] Oct 31 23:07:56.453414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.453383 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs54c\" (UniqueName: \"kubernetes.io/projected/6983aade-db19-4509-b5d1-dc7c81f8bb88-kube-api-access-bs54c\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.453572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.453508 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6983aade-db19-4509-b5d1-dc7c81f8bb88-metrics-certs\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.554151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.554114 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6983aade-db19-4509-b5d1-dc7c81f8bb88-metrics-certs\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.554334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.554177 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs54c\" (UniqueName: \"kubernetes.io/projected/6983aade-db19-4509-b5d1-dc7c81f8bb88-kube-api-access-bs54c\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.556621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.556593 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6983aade-db19-4509-b5d1-dc7c81f8bb88-metrics-certs\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.577401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.577376 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs54c\" (UniqueName: \"kubernetes.io/projected/6983aade-db19-4509-b5d1-dc7c81f8bb88-kube-api-access-bs54c\") pod \"ai-gateway-controller-7d58dbf686-blgqt\" (UID: \"6983aade-db19-4509-b5d1-dc7c81f8bb88\") " pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.723710 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.723624 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:56.998354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:56.998329 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d58dbf686-blgqt"] Oct 31 23:07:57.000112 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:07:57.000082 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6983aade_db19_4509_b5d1_dc7c81f8bb88.slice/crio-f68b08549da9fcb569baa137c93245b458895e9d7955277ea737e2e7e4f2828f WatchSource:0}: Error finding container f68b08549da9fcb569baa137c93245b458895e9d7955277ea737e2e7e4f2828f: Status 404 returned error can't find the container with id f68b08549da9fcb569baa137c93245b458895e9d7955277ea737e2e7e4f2828f Oct 31 23:07:57.223208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:57.223167 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" event={"ID":"6983aade-db19-4509-b5d1-dc7c81f8bb88","Type":"ContainerStarted","Data":"f68b08549da9fcb569baa137c93245b458895e9d7955277ea737e2e7e4f2828f"} Oct 31 23:07:58.230066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:58.230032 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" event={"ID":"6983aade-db19-4509-b5d1-dc7c81f8bb88","Type":"ContainerStarted","Data":"2633314dd6d573d85f4330d26db90354a2d0bbd2ba4294f95d88177d1a1d0955"} Oct 31 23:07:58.230546 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:58.230112 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:07:58.251018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:07:58.250954 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" podStartSLOduration=1.8380018 podStartE2EDuration="2.25094212s" totalImagesPullingTime="412.94032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:58.250186986 +0000 UTC m=+1056.232929363" watchObservedRunningTime="2026-10-31 23:07:58.25094212 +0000 UTC m=+1056.233684497" Oct 31 23:08:01.195753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:01.195726 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xdxrp" Oct 31 23:08:01.196201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:01.195791 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-7ql7n" Oct 31 23:08:02.272289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.272254 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:08:02.275964 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.275947 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.291017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.290970 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:08:02.301348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301313 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96shw\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301356 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301399 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301469 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301624 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301649 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301729 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301679 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301831 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301813 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.301879 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.301862 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.311450 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.311422 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:08:02.315163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.315144 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.344715 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.344686 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:08:02.403259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403230 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403266 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96shw\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403288 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403310 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403332 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403432 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403470 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403515 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403596 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403619 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403685 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403765 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403835 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.403901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403867 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.403983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403932 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.404042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.403984 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.404099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404050 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.404099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404075 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.404099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404086 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.404262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404100 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hn5l\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.404318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404286 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.404421 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404399 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.404906 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.404882 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.405814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.405797 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.405946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.405928 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.415858 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.415835 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.415982 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.415916 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96shw\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\") pod \"maas-default-gateway-openshift-default-6886b96f9-p76fd\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.505474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505433 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505491 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505512 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505540 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505587 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505607 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505631 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hn5l\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505690 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.505885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505725 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.506066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505909 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.506066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.505918 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.506160 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.506095 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.506228 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.506213 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.506295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.506267 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.508051 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.508029 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.508167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.508114 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.518350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.518326 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.518674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.518656 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hn5l\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\") pod \"maas-default-gateway-openshift-default-6886b96f9-mcdvp\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.590285 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.590205 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:02.628524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.628490 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:02.874009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.873973 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:08:02.876275 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:02.876251 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a4b5f6_7b12_4e65_a63c_88d7c06c2c74.slice/crio-c8cd0dc2604a922ddb2cbbf28406997c96b4518ae7db8955caf40f9c2663309c WatchSource:0}: Error finding container c8cd0dc2604a922ddb2cbbf28406997c96b4518ae7db8955caf40f9c2663309c: Status 404 returned error can't find the container with id c8cd0dc2604a922ddb2cbbf28406997c96b4518ae7db8955caf40f9c2663309c Oct 31 23:08:02.878493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.878461 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:02.878575 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.878537 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:02.878641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.878584 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:02.927753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.927730 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:08:02.930074 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:02.930046 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c529ca5_f36e_4c77_a315_e5ffa4609612.slice/crio-bdcbd3667ddcdfbda42c239fcc475b964164a5d4e13dec883ac95c0920359e2a WatchSource:0}: Error finding container bdcbd3667ddcdfbda42c239fcc475b964164a5d4e13dec883ac95c0920359e2a: Status 404 returned error can't find the container with id bdcbd3667ddcdfbda42c239fcc475b964164a5d4e13dec883ac95c0920359e2a Oct 31 23:08:02.932336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.932303 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:02.932405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.932368 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:02.932450 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:02.932401 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:08:03.260448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.260412 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" event={"ID":"2c529ca5-f36e-4c77-a315-e5ffa4609612","Type":"ContainerStarted","Data":"3654dca51588c088a8d1c1810d2c52532d8c536a0ca68e79c39fc99aefcc2000"} Oct 31 23:08:03.260646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.260472 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" event={"ID":"2c529ca5-f36e-4c77-a315-e5ffa4609612","Type":"ContainerStarted","Data":"bdcbd3667ddcdfbda42c239fcc475b964164a5d4e13dec883ac95c0920359e2a"} Oct 31 23:08:03.262248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.262223 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" event={"ID":"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74","Type":"ContainerStarted","Data":"63fa1df26e9728d0a5ddca850afc17c30cab3803c1bc6e84c985f9f3c6021880"} Oct 31 23:08:03.262381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.262264 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" event={"ID":"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74","Type":"ContainerStarted","Data":"c8cd0dc2604a922ddb2cbbf28406997c96b4518ae7db8955caf40f9c2663309c"} Oct 31 23:08:03.287897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.287021 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podStartSLOduration=1.286979773 podStartE2EDuration="1.286979773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:03.285487042 +0000 UTC m=+1061.268229431" watchObservedRunningTime="2026-10-31 23:08:03.286979773 +0000 UTC m=+1061.269722151" Oct 31 23:08:03.312272 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.312211 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podStartSLOduration=1.312190966 podStartE2EDuration="1.312190966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:03.309283807 +0000 UTC m=+1061.292026186" watchObservedRunningTime="2026-10-31 23:08:03.312190966 +0000 UTC m=+1061.294933347" Oct 31 23:08:03.594144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.593365 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:03.595140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.595112 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:03.595273 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.595166 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:03.629454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:03.629416 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:04.592415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:04.592377 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:04.593015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:04.592957 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:04.631045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:04.631012 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:04.631166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:04.631060 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:05.592411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:05.592381 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:05.592662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:05.592640 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:05.629087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:05.629046 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:05.629191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:05.629095 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:06.593034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:06.592446 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:06.593034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:06.592514 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:06.629533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:06.629499 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:06.629735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:06.629557 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:07.591473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:07.591436 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:07.591801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:07.591769 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:07.629170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:07.629138 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:07.629604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:07.629576 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:08.592525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:08.592478 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:08.592717 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:08.592553 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:08.629095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:08.629055 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:08.629260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:08.629106 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:09.238905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.238876 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d58dbf686-blgqt" Oct 31 23:08:09.288338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.288135 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:08:09.288527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.288430 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" podUID="f847ea5e-71be-435b-b177-775b4ebd9850" containerName="manager" containerID="cri-o://9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd" gracePeriod=30 Oct 31 23:08:09.547775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.547745 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:08:09.591078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.591037 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:09.591238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.591107 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 23:08:09.627729 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.627692 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\") pod \"f847ea5e-71be-435b-b177-775b4ebd9850\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " Oct 31 23:08:09.627887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.627865 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\" (UniqueName: \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\") pod \"f847ea5e-71be-435b-b177-775b4ebd9850\" (UID: \"f847ea5e-71be-435b-b177-775b4ebd9850\") " Oct 31 23:08:09.628943 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.628915 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:09.629081 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.628982 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 23:08:09.630232 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.630204 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs" pod "f847ea5e-71be-435b-b177-775b4ebd9850" (UID: "f847ea5e-71be-435b-b177-775b4ebd9850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:08:09.630597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.630280 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx" pod "f847ea5e-71be-435b-b177-775b4ebd9850" (UID: "f847ea5e-71be-435b-b177-775b4ebd9850"). InnerVolumeSpecName "kube-api-access-526tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:08:09.728978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.728938 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f847ea5e-71be-435b-b177-775b4ebd9850-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:08:09.728978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:09.728971 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526tx\" (UniqueName: \"kubernetes.io/projected/f847ea5e-71be-435b-b177-775b4ebd9850-kube-api-access-526tx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:08:10.328915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.328883 7074 generic.go:356] "Generic (PLEG): container finished" podID="f847ea5e-71be-435b-b177-775b4ebd9850" containerID="9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd" exitCode=0 Oct 31 23:08:10.329113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.328959 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" Oct 31 23:08:10.329113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.328971 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" event={"ID":"f847ea5e-71be-435b-b177-775b4ebd9850","Type":"ContainerDied","Data":"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd"} Oct 31 23:08:10.329113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.329057 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-28vck" event={"ID":"f847ea5e-71be-435b-b177-775b4ebd9850","Type":"ContainerDied","Data":"b48bff13b9a22b7ea2f5ef4b9626f611dcd90555cb9f492e0df1f6e474e8e082"} Oct 31 23:08:10.329113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.329089 7074 scope.go:122] "RemoveContainer" containerID="9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd" Oct 31 23:08:10.339888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.339869 7074 scope.go:122] "RemoveContainer" containerID="9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd" Oct 31 23:08:10.340222 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:08:10.340198 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd\": container with ID starting with 9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd not found: ID does not exist" containerID="9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd" Oct 31 23:08:10.340285 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.340237 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd"} err="failed to get container status \"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd\": rpc error: code = NotFound desc = could not find container \"9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd\": container with ID starting with 9dd0214c9b9d884e34d5bb1cb0f4021e51f68ba39fa9367331e290a62b3d16dd not found: ID does not exist" Oct 31 23:08:10.350134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.350114 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:08:10.354777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:10.354754 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-28vck"] Oct 31 23:08:11.257249 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.257212 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:11.257725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.257548 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:11.258433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.258413 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:08:11.287854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.287824 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:11.288157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.288136 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:11.288974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:11.288938 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:08:12.141404 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:12.141371 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f847ea5e-71be-435b-b177-775b4ebd9850" path="/var/lib/kubelet/pods/f847ea5e-71be-435b-b177-775b4ebd9850/volumes" Oct 31 23:08:14.887577 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:08:14.887524 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:08:44.261732 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.261699 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:08:44.262298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.262195 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f847ea5e-71be-435b-b177-775b4ebd9850" containerName="manager" Oct 31 23:08:44.262298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.262209 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f847ea5e-71be-435b-b177-775b4ebd9850" containerName="manager" Oct 31 23:08:44.262298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.262297 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f847ea5e-71be-435b-b177-775b4ebd9850" containerName="manager" Oct 31 23:08:44.265418 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.265403 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.277150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.277125 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-135248-dockercfg-fh69f\"" Oct 31 23:08:44.277719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.277699 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-135248\"" Oct 31 23:08:44.282045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.282015 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:08:44.306722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.306680 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:08:44.310416 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.310391 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.340511 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.340476 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:08:44.389113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.389073 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fxl\" (UniqueName: \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.389327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.389149 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.389327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.389189 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdln\" (UniqueName: \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.389327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.389303 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.491039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.490981 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fxl\" (UniqueName: \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.491245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.491097 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.491245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.491133 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdln\" (UniqueName: \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.491245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.491175 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.491964 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.491940 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.492219 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.492196 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.505591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.505559 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fxl\" (UniqueName: \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\") pod \"payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.505591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.505586 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdln\" (UniqueName: \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\") pod \"payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.579641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.579512 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:44.625314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.625280 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:44.866266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.866229 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:08:44.868578 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:44.868545 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46131fa4_7daa_4a14_bda7_dcc36fc6eedb.slice/crio-d5c2162b399ec7ea367350c10d94d062d9bdf888384e85707ea3df2be2065279 WatchSource:0}: Error finding container d5c2162b399ec7ea367350c10d94d062d9bdf888384e85707ea3df2be2065279: Status 404 returned error can't find the container with id d5c2162b399ec7ea367350c10d94d062d9bdf888384e85707ea3df2be2065279 Oct 31 23:08:44.899567 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:08:44.899509 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:08:44.930408 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.930383 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:08:44.932455 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:44.932420 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c96086e_8783_457f_9aa5_dd76960b42b5.slice/crio-d0fdbe5f9f847c60d9816d33dbf80f93ccbd1049db2d3cccf1ee5a3be29ad446 WatchSource:0}: Error finding container d0fdbe5f9f847c60d9816d33dbf80f93ccbd1049db2d3cccf1ee5a3be29ad446: Status 404 returned error can't find the container with id d0fdbe5f9f847c60d9816d33dbf80f93ccbd1049db2d3cccf1ee5a3be29ad446 Oct 31 23:08:44.980070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.980048 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:08:44.985082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.985063 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:44.994133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.992749 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-255748\"" Oct 31 23:08:44.994133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.993270 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-255748-dockercfg-9vqm4\"" Oct 31 23:08:44.995647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:44.995507 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:08:45.030212 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.030180 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:08:45.033970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.033950 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.044538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.044508 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:08:45.100965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.100931 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.101235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.101092 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwmg\" (UniqueName: \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.101235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.101136 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn86f\" (UniqueName: \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.101235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.101188 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.202572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.202494 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwmg\" (UniqueName: \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.202572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.202558 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn86f\" (UniqueName: \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.202809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.202626 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.202809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.202697 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.203409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.203386 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.203527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.203438 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.223796 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.223763 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn86f\" (UniqueName: \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\") pod \"payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.223958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.223893 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:08:45.224413 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.224391 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwmg\" (UniqueName: \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\") pod \"payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.229292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.229265 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.233724 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.233700 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\"" Oct 31 23:08:45.233859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.233798 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-135248-serving-cert\"" Oct 31 23:08:45.233859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.233848 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 23:08:45.242021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.240400 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:08:45.303845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.303809 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:45.304302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.304180 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.304302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.304237 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfpj\" (UniqueName: \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.304381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.304323 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.304417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.304394 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.354731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.354695 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:45.407150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.405881 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.407150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.405943 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfpj\" (UniqueName: \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.407150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.405984 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.407150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.406054 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.407150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.407001 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.411220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.411184 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.413215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.413184 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.420745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.420627 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfpj\" (UniqueName: \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\") pod \"maas-api-e2e-worker-w4-135248-f745d847d-fgg8v\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.538039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.537980 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" event={"ID":"6c96086e-8783-457f-9aa5-dd76960b42b5","Type":"ContainerStarted","Data":"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc"} Oct 31 23:08:45.538211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.538067 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" event={"ID":"6c96086e-8783-457f-9aa5-dd76960b42b5","Type":"ContainerStarted","Data":"d0fdbe5f9f847c60d9816d33dbf80f93ccbd1049db2d3cccf1ee5a3be29ad446"} Oct 31 23:08:45.538211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.538186 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:45.539569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.539541 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" event={"ID":"46131fa4-7daa-4a14-bda7-dcc36fc6eedb","Type":"ContainerStarted","Data":"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d"} Oct 31 23:08:45.539700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.539603 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:45.539700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.539624 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" event={"ID":"46131fa4-7daa-4a14-bda7-dcc36fc6eedb","Type":"ContainerStarted","Data":"d5c2162b399ec7ea367350c10d94d062d9bdf888384e85707ea3df2be2065279"} Oct 31 23:08:45.545771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.545753 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:45.558627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.558573 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" podStartSLOduration=1.558556715 podStartE2EDuration="1.558556715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:45.556546796 +0000 UTC m=+1103.539289198" watchObservedRunningTime="2026-10-31 23:08:45.558556715 +0000 UTC m=+1103.541299093" Oct 31 23:08:45.584974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.583236 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" podStartSLOduration=1.583220368 podStartE2EDuration="1.583220368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:45.580019184 +0000 UTC m=+1103.562761554" watchObservedRunningTime="2026-10-31 23:08:45.583220368 +0000 UTC m=+1103.565962744" Oct 31 23:08:45.585722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.585690 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:08:45.590159 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:45.590129 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd605101_22ca_4801_bc41_2f734374bc8f.slice/crio-ffea93880a501204b45dad136ad8714d84201437e8d15e52bf63364ad118972f WatchSource:0}: Error finding container ffea93880a501204b45dad136ad8714d84201437e8d15e52bf63364ad118972f: Status 404 returned error can't find the container with id ffea93880a501204b45dad136ad8714d84201437e8d15e52bf63364ad118972f Oct 31 23:08:45.654650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.654620 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:08:45.656124 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:45.656084 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb787ffbb_5157_475b_8eb2_b7c6f5e8671d.slice/crio-4a37bd0bdee735649e843673126d926b7c87716e5fe7a80e9e914a5f3ec0f1b3 WatchSource:0}: Error finding container 4a37bd0bdee735649e843673126d926b7c87716e5fe7a80e9e914a5f3ec0f1b3: Status 404 returned error can't find the container with id 4a37bd0bdee735649e843673126d926b7c87716e5fe7a80e9e914a5f3ec0f1b3 Oct 31 23:08:45.826266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.826193 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:08:45.832198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.832168 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:45.838449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.838189 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:08:45.840632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.840608 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a16dfb\"" Oct 31 23:08:45.841900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.841874 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:08:45.842161 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:45.842127 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db814a8_8d7b_40cb_b938_788f64abf26a.slice/crio-acf8fb1973cc1d5ead63412dcb9ad89db5efc0fd6513d5453ef35966fcd9dd20 WatchSource:0}: Error finding container acf8fb1973cc1d5ead63412dcb9ad89db5efc0fd6513d5453ef35966fcd9dd20: Status 404 returned error can't find the container with id acf8fb1973cc1d5ead63412dcb9ad89db5efc0fd6513d5453ef35966fcd9dd20 Oct 31 23:08:45.858716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.858689 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:08:45.862443 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.862426 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:45.872923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.872881 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:08:45.911188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.911154 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:45.911362 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.911292 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgc4m\" (UniqueName: \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:45.920117 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:45.920092 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a16dfb-dockercfg-mh4mt\"" Oct 31 23:08:46.012079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.012042 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.012269 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.012173 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgc4m\" (UniqueName: \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.012269 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.012208 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.012269 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.012244 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkcv\" (UniqueName: \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.012940 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.012920 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.035242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.035176 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgc4m\" (UniqueName: \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\") pod \"payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.113833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.113730 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.113833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.113776 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkcv\" (UniqueName: \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.114558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.114525 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.127376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.127337 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkcv\" (UniqueName: \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\") pod \"payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.156682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.156645 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.171362 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.171337 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:08:46.176319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.176296 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.176838 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.176809 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:46.182494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.182472 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-04c5c2-dockercfg-zgndh\"" Oct 31 23:08:46.185775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.185518 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:08:46.187164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.187136 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-04c5c2\"" Oct 31 23:08:46.210934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.210907 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:08:46.217022 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.216982 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.242044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.241951 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:08:46.333295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.326688 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwgw\" (UniqueName: \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.334165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.334140 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.334294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.334265 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.334358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.334317 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6cs\" (UniqueName: \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.435062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435039 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.435138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435096 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.435138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435123 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6cs\" (UniqueName: \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.435231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435200 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwgw\" (UniqueName: \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.435821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435799 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.435911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.435898 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.440460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.440439 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:08:46.442864 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:46.442840 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30089af_389e_4d7f_8681_9fde7bfeba30.slice/crio-bae055892a3f8b29d02db7af6ac2b2956bc0595d7bab60ee3388ad737ac7c7a7 WatchSource:0}: Error finding container bae055892a3f8b29d02db7af6ac2b2956bc0595d7bab60ee3388ad737ac7c7a7: Status 404 returned error can't find the container with id bae055892a3f8b29d02db7af6ac2b2956bc0595d7bab60ee3388ad737ac7c7a7 Oct 31 23:08:46.451267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.451244 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwgw\" (UniqueName: \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\") pod \"payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.451898 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.451877 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6cs\" (UniqueName: \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\") pod \"payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.494091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.494062 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:46.510482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.510459 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:08:46.514110 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:46.514082 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148d362e_8159_43a4_890a_c049be03ac0e.slice/crio-1518b993170114070b0d1fa61052b4f26f0ab0078c33d51cc0865d1297733296 WatchSource:0}: Error finding container 1518b993170114070b0d1fa61052b4f26f0ab0078c33d51cc0865d1297733296: Status 404 returned error can't find the container with id 1518b993170114070b0d1fa61052b4f26f0ab0078c33d51cc0865d1297733296 Oct 31 23:08:46.544603 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.544380 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:46.552374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.552349 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" event={"ID":"6db814a8-8d7b-40cb-b938-788f64abf26a","Type":"ContainerStarted","Data":"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7"} Oct 31 23:08:46.552531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.552519 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" event={"ID":"6db814a8-8d7b-40cb-b938-788f64abf26a","Type":"ContainerStarted","Data":"acf8fb1973cc1d5ead63412dcb9ad89db5efc0fd6513d5453ef35966fcd9dd20"} Oct 31 23:08:46.552716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.552699 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:46.557265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.556164 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" event={"ID":"dd605101-22ca-4801-bc41-2f734374bc8f","Type":"ContainerStarted","Data":"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb"} Oct 31 23:08:46.557265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.556220 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" event={"ID":"dd605101-22ca-4801-bc41-2f734374bc8f","Type":"ContainerStarted","Data":"ffea93880a501204b45dad136ad8714d84201437e8d15e52bf63364ad118972f"} Oct 31 23:08:46.557265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.556836 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:46.569953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.569094 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" event={"ID":"b787ffbb-5157-475b-8eb2-b7c6f5e8671d","Type":"ContainerStarted","Data":"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485"} Oct 31 23:08:46.569953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.569155 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" event={"ID":"b787ffbb-5157-475b-8eb2-b7c6f5e8671d","Type":"ContainerStarted","Data":"4a37bd0bdee735649e843673126d926b7c87716e5fe7a80e9e914a5f3ec0f1b3"} Oct 31 23:08:46.569953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.569896 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:46.576063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.575871 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" event={"ID":"a30089af-389e-4d7f-8681-9fde7bfeba30","Type":"ContainerStarted","Data":"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d"} Oct 31 23:08:46.576063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.575926 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" event={"ID":"a30089af-389e-4d7f-8681-9fde7bfeba30","Type":"ContainerStarted","Data":"bae055892a3f8b29d02db7af6ac2b2956bc0595d7bab60ee3388ad737ac7c7a7"} Oct 31 23:08:46.578054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.578034 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:46.581315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.581291 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" event={"ID":"148d362e-8159-43a4-890a-c049be03ac0e","Type":"ContainerStarted","Data":"1518b993170114070b0d1fa61052b4f26f0ab0078c33d51cc0865d1297733296"} Oct 31 23:08:46.584626 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.584577 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" podStartSLOduration=1.055519524 podStartE2EDuration="1.584561743s" totalImagesPullingTime="529.042219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:46.582490286 +0000 UTC m=+1104.565232662" watchObservedRunningTime="2026-10-31 23:08:46.584561743 +0000 UTC m=+1104.567304122" Oct 31 23:08:46.634812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.634492 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" podStartSLOduration=2.63447661 podStartE2EDuration="2.63447661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:46.632196628 +0000 UTC m=+1104.614939005" watchObservedRunningTime="2026-10-31 23:08:46.63447661 +0000 UTC m=+1104.617218986" Oct 31 23:08:46.635045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.634771 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" podStartSLOduration=1.634764927 podStartE2EDuration="1.634764927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:46.611579652 +0000 UTC m=+1104.594322032" watchObservedRunningTime="2026-10-31 23:08:46.634764927 +0000 UTC m=+1104.617507303" Oct 31 23:08:46.660564 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.660500 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" podStartSLOduration=1.660478127 podStartE2EDuration="1.660478127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:46.65841215 +0000 UTC m=+1104.641154526" watchObservedRunningTime="2026-10-31 23:08:46.660478127 +0000 UTC m=+1104.643220505" Oct 31 23:08:46.757766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.757723 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:08:46.762064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.762040 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.772465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.772428 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:08:46.786209 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.786174 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:08:46.791443 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:46.791417 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda779def2_64c5_4553_b53a_0ff1564605ab.slice/crio-045848c6e028ff8ead085bf12e6ea797508c1b02d557dc1ffe8dab4a55a8f12a WatchSource:0}: Error finding container 045848c6e028ff8ead085bf12e6ea797508c1b02d557dc1ffe8dab4a55a8f12a: Status 404 returned error can't find the container with id 045848c6e028ff8ead085bf12e6ea797508c1b02d557dc1ffe8dab4a55a8f12a Oct 31 23:08:46.791585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.791530 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:08:46.792802 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.792759 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-67fce9\"" Oct 31 23:08:46.795975 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.795955 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:46.807537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.807310 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:08:46.811355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.811333 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-67fce9-dockercfg-9gxqn\"" Oct 31 23:08:46.840366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.840335 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55cp\" (UniqueName: \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.840503 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.840387 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.872472 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.872439 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:08:46.876414 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:46.876386 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc3108_a881_484a_bda6_ccc8f9374bbe.slice/crio-54d77c652439ceb5095476b89c5d8ce4646dcb3edd9de8f4fe5202108962ba42 WatchSource:0}: Error finding container 54d77c652439ceb5095476b89c5d8ce4646dcb3edd9de8f4fe5202108962ba42: Status 404 returned error can't find the container with id 54d77c652439ceb5095476b89c5d8ce4646dcb3edd9de8f4fe5202108962ba42 Oct 31 23:08:46.941470 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.941439 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55cp\" (UniqueName: \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.941649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.941486 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.941649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.941526 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:46.941649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.941607 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbncb\" (UniqueName: \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:46.942184 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.942162 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:46.955397 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:46.955367 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55cp\" (UniqueName: \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\") pod \"payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:47.001278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.001246 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:08:47.005562 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.005534 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.009769 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.009748 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-255748-serving-cert\"" Oct 31 23:08:47.020870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.020843 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:08:47.043198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.043161 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbncb\" (UniqueName: \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.043363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.043291 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.044229 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.044202 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.056304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.056271 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbncb\" (UniqueName: \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\") pod \"payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.079109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.079073 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:47.122922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.122887 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.145324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.145212 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.145604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.145576 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg92\" (UniqueName: \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.145722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.145650 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.145722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.145683 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.246638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.246590 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg92\" (UniqueName: \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.246811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.246680 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.246811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.246715 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.246924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.246824 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.247403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.247380 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.249785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.249765 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.249953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.249926 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.258871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.258852 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg92\" (UniqueName: \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\") pod \"maas-api-e2e-worker-w2-255748-85bb494466-sv4dk\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.324020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.323981 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:47.375590 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:47.375543 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e0bf96_8ee5_4334_a767_f69061825885.slice/crio-4edc3ab7f1c1a708f8dd507ba5ce089b307ca0d30f9424f73495fb95cb80caf8 WatchSource:0}: Error finding container 4edc3ab7f1c1a708f8dd507ba5ce089b307ca0d30f9424f73495fb95cb80caf8: Status 404 returned error can't find the container with id 4edc3ab7f1c1a708f8dd507ba5ce089b307ca0d30f9424f73495fb95cb80caf8 Oct 31 23:08:47.376028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.375752 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:08:47.419674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.419644 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:08:47.431576 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:47.430084 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00fd71e_9c7a_4b34_8bd2_cad937a6ab04.slice/crio-f8cfcc65e6942dd55a44a626c40e531c7b519eb6f6a02965c6bf29bd7c582d6a WatchSource:0}: Error finding container f8cfcc65e6942dd55a44a626c40e531c7b519eb6f6a02965c6bf29bd7c582d6a: Status 404 returned error can't find the container with id f8cfcc65e6942dd55a44a626c40e531c7b519eb6f6a02965c6bf29bd7c582d6a Oct 31 23:08:47.536974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.536938 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:08:47.541256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.541236 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.552240 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.552214 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2294a7\"" Oct 31 23:08:47.552426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.552403 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:08:47.563205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.563175 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:08:47.563354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.563292 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.568085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.568061 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2294a7-dockercfg-ntfv8\"" Oct 31 23:08:47.572555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.572461 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:08:47.592532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.592502 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" event={"ID":"a779def2-64c5-4553-b53a-0ff1564605ab","Type":"ContainerStarted","Data":"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416"} Oct 31 23:08:47.592695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.592557 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" event={"ID":"a779def2-64c5-4553-b53a-0ff1564605ab","Type":"ContainerStarted","Data":"045848c6e028ff8ead085bf12e6ea797508c1b02d557dc1ffe8dab4a55a8f12a"} Oct 31 23:08:47.592756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.592695 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:47.594582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.594554 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" event={"ID":"148d362e-8159-43a4-890a-c049be03ac0e","Type":"ContainerStarted","Data":"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489"} Oct 31 23:08:47.594722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.594621 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:47.596530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.596507 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" event={"ID":"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04","Type":"ContainerStarted","Data":"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b"} Oct 31 23:08:47.596613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.596551 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" event={"ID":"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04","Type":"ContainerStarted","Data":"f8cfcc65e6942dd55a44a626c40e531c7b519eb6f6a02965c6bf29bd7c582d6a"} Oct 31 23:08:47.596698 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.596686 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:47.598466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.598443 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" event={"ID":"17e0bf96-8ee5-4334-a767-f69061825885","Type":"ContainerStarted","Data":"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc"} Oct 31 23:08:47.598558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.598488 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" event={"ID":"17e0bf96-8ee5-4334-a767-f69061825885","Type":"ContainerStarted","Data":"4edc3ab7f1c1a708f8dd507ba5ce089b307ca0d30f9424f73495fb95cb80caf8"} Oct 31 23:08:47.598558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.598515 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:47.600262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.600239 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" event={"ID":"80dc3108-a881-484a-bda6-ccc8f9374bbe","Type":"ContainerStarted","Data":"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1"} Oct 31 23:08:47.600357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.600277 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" event={"ID":"80dc3108-a881-484a-bda6-ccc8f9374bbe","Type":"ContainerStarted","Data":"54d77c652439ceb5095476b89c5d8ce4646dcb3edd9de8f4fe5202108962ba42"} Oct 31 23:08:47.600435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.600421 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:47.611764 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.611742 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:08:47.613238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.613196 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" podStartSLOduration=1.6131846269999999 podStartE2EDuration="1.613184627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:47.611928644 +0000 UTC m=+1105.594671023" watchObservedRunningTime="2026-10-31 23:08:47.613184627 +0000 UTC m=+1105.595927004" Oct 31 23:08:47.615494 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:47.615471 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9352f587_17c5_4cb8_a7b3_0e5a517e882f.slice/crio-fac635448d01d0d2abeb6118974b2bec385038ce5529982fe3036741d5505754 WatchSource:0}: Error finding container fac635448d01d0d2abeb6118974b2bec385038ce5529982fe3036741d5505754: Status 404 returned error can't find the container with id fac635448d01d0d2abeb6118974b2bec385038ce5529982fe3036741d5505754 Oct 31 23:08:47.633900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.633837 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" podStartSLOduration=1.633826962 podStartE2EDuration="1.633826962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:47.631021856 +0000 UTC m=+1105.613764229" watchObservedRunningTime="2026-10-31 23:08:47.633826962 +0000 UTC m=+1105.616569339" Oct 31 23:08:47.656421 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.656373 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" podStartSLOduration=2.65636011 podStartE2EDuration="2.65636011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:47.654891661 +0000 UTC m=+1105.637634039" watchObservedRunningTime="2026-10-31 23:08:47.65636011 +0000 UTC m=+1105.639102487" Oct 31 23:08:47.657176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.657148 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnf5\" (UniqueName: \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.657176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.657192 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.657383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.657262 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.657440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.657390 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gzh\" (UniqueName: \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.674259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.674217 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" podStartSLOduration=1.674207808 podStartE2EDuration="1.674207808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:47.673662956 +0000 UTC m=+1105.656405333" watchObservedRunningTime="2026-10-31 23:08:47.674207808 +0000 UTC m=+1105.656950184" Oct 31 23:08:47.695813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.695760 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" podStartSLOduration=1.695745238 podStartE2EDuration="1.695745238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:47.692554001 +0000 UTC m=+1105.675296389" watchObservedRunningTime="2026-10-31 23:08:47.695745238 +0000 UTC m=+1105.678487619" Oct 31 23:08:47.759357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.759053 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gzh\" (UniqueName: \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.759357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.759124 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnf5\" (UniqueName: \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.759357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.759155 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.759635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.759411 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.760193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.760162 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.760487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.760466 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.771714 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.771692 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gzh\" (UniqueName: \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\") pod \"payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.771844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.771823 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnf5\" (UniqueName: \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\") pod \"payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.819899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.819848 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:08:47.826355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.826327 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:47.832178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.832153 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a16dfb-serving-cert\"" Oct 31 23:08:47.844270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.844194 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:08:47.866226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.866191 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:47.882035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.881972 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:47.967839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.967810 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:47.967959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.967870 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:47.967959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.967911 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwmq\" (UniqueName: \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:47.968220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:47.968199 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.068921 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.068892 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.069078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.068966 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.069078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.069057 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.069265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.069100 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwmq\" (UniqueName: \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.069754 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.069728 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.071385 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.071364 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.071801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.071780 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.090248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.090216 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwmq\" (UniqueName: \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\") pod \"maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.161321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.161252 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:48.166981 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.166956 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:08:48.168316 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:48.168290 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cf81c8_dfcb_4c44_92fa_a9033644e2c3.slice/crio-7cd1ffb65f7d1896efee70fcd2791cab4de62b5acc5b3179574efdcabcd8b0ec WatchSource:0}: Error finding container 7cd1ffb65f7d1896efee70fcd2791cab4de62b5acc5b3179574efdcabcd8b0ec: Status 404 returned error can't find the container with id 7cd1ffb65f7d1896efee70fcd2791cab4de62b5acc5b3179574efdcabcd8b0ec Oct 31 23:08:48.184839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.184813 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:08:48.189149 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:48.189108 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514f3f01_5825_420b_93f1_8e8a294ccb70.slice/crio-727e1ff4c1da5d6d616289a3819fd131afe2f43c1136a8456e59028a8755578b WatchSource:0}: Error finding container 727e1ff4c1da5d6d616289a3819fd131afe2f43c1136a8456e59028a8755578b: Status 404 returned error can't find the container with id 727e1ff4c1da5d6d616289a3819fd131afe2f43c1136a8456e59028a8755578b Oct 31 23:08:48.479953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.479926 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:08:48.481790 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:48.481765 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ef6af7_eb72_45dc_9106_ef21889ab2f2.slice/crio-c8f8e20468dbbf11d663c2d8c3d4ab6a3e898757db997932e3095d777598016e WatchSource:0}: Error finding container c8f8e20468dbbf11d663c2d8c3d4ab6a3e898757db997932e3095d777598016e: Status 404 returned error can't find the container with id c8f8e20468dbbf11d663c2d8c3d4ab6a3e898757db997932e3095d777598016e Oct 31 23:08:48.616355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.616313 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" event={"ID":"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3","Type":"ContainerStarted","Data":"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b"} Oct 31 23:08:48.616537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.616385 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" event={"ID":"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3","Type":"ContainerStarted","Data":"7cd1ffb65f7d1896efee70fcd2791cab4de62b5acc5b3179574efdcabcd8b0ec"} Oct 31 23:08:48.616537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.616418 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:08:48.618299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.618270 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" event={"ID":"69ef6af7-eb72-45dc-9106-ef21889ab2f2","Type":"ContainerStarted","Data":"c8f8e20468dbbf11d663c2d8c3d4ab6a3e898757db997932e3095d777598016e"} Oct 31 23:08:48.620179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.620153 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" event={"ID":"9352f587-17c5-4cb8-a7b3-0e5a517e882f","Type":"ContainerStarted","Data":"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378"} Oct 31 23:08:48.620313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.620204 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" event={"ID":"9352f587-17c5-4cb8-a7b3-0e5a517e882f","Type":"ContainerStarted","Data":"fac635448d01d0d2abeb6118974b2bec385038ce5529982fe3036741d5505754"} Oct 31 23:08:48.620313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.620236 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:48.621801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.621780 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" event={"ID":"514f3f01-5825-420b-93f1-8e8a294ccb70","Type":"ContainerStarted","Data":"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442"} Oct 31 23:08:48.621905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.621822 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" event={"ID":"514f3f01-5825-420b-93f1-8e8a294ccb70","Type":"ContainerStarted","Data":"727e1ff4c1da5d6d616289a3819fd131afe2f43c1136a8456e59028a8755578b"} Oct 31 23:08:48.638863 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.638803 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" podStartSLOduration=1.6387891049999999 podStartE2EDuration="1.638789105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:48.637198225 +0000 UTC m=+1106.619940604" watchObservedRunningTime="2026-10-31 23:08:48.638789105 +0000 UTC m=+1106.621531518" Oct 31 23:08:48.659597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.659530 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" podStartSLOduration=2.250112795 podStartE2EDuration="2.659509734s" totalImagesPullingTime="409.396939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:48.658304361 +0000 UTC m=+1106.641046763" watchObservedRunningTime="2026-10-31 23:08:48.659509734 +0000 UTC m=+1106.642252110" Oct 31 23:08:48.679374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.679319 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" podStartSLOduration=1.679304669 podStartE2EDuration="1.679304669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:48.676511893 +0000 UTC m=+1106.659254295" watchObservedRunningTime="2026-10-31 23:08:48.679304669 +0000 UTC m=+1106.662047044" Oct 31 23:08:48.831386 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.831353 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:08:48.837607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.837582 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:48.854821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.854787 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:08:48.856484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.856459 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-04c5c2-serving-cert\"" Oct 31 23:08:48.980440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.980405 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxqt\" (UniqueName: \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:48.980604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.980558 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:48.980678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.980649 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:48.980726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:48.980712 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.082414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.082330 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxqt\" (UniqueName: \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.082590 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.082440 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.082590 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.082461 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.082590 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.082486 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.083222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.083191 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.085058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.085038 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.085058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.085057 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.095061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.095037 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxqt\" (UniqueName: \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\") pod \"maas-api-e2e-worker-w1-04c5c2-857d954885-24g95\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.159205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.159175 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:49.442347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.442319 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:08:49.444013 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:49.443966 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8db220b_9983_4da1_b152_6bc107121c17.slice/crio-6336ecb0b7e7cb68e3f883a1c8a529cadffaeb3e5ac7504244b2ac2bce690823 WatchSource:0}: Error finding container 6336ecb0b7e7cb68e3f883a1c8a529cadffaeb3e5ac7504244b2ac2bce690823: Status 404 returned error can't find the container with id 6336ecb0b7e7cb68e3f883a1c8a529cadffaeb3e5ac7504244b2ac2bce690823 Oct 31 23:08:49.630155 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.630120 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" event={"ID":"b8db220b-9983-4da1-b152-6bc107121c17","Type":"ContainerStarted","Data":"6336ecb0b7e7cb68e3f883a1c8a529cadffaeb3e5ac7504244b2ac2bce690823"} Oct 31 23:08:49.631965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.631931 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" event={"ID":"69ef6af7-eb72-45dc-9106-ef21889ab2f2","Type":"ContainerStarted","Data":"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15"} Oct 31 23:08:49.632137 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.632041 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:49.632579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.632557 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:08:49.656407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:49.656351 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" podStartSLOduration=2.204267339 podStartE2EDuration="2.656335175s" totalImagesPullingTime="452.067836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:49.65497571 +0000 UTC m=+1107.637718090" watchObservedRunningTime="2026-10-31 23:08:49.656335175 +0000 UTC m=+1107.639077553" Oct 31 23:08:50.648140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.648106 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" event={"ID":"b8db220b-9983-4da1-b152-6bc107121c17","Type":"ContainerStarted","Data":"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099"} Oct 31 23:08:50.648550 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.648215 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:50.675341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.675291 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" podStartSLOduration=2.201228944 podStartE2EDuration="2.675277092s" totalImagesPullingTime="474.048148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:50.672526431 +0000 UTC m=+1108.655268809" watchObservedRunningTime="2026-10-31 23:08:50.675277092 +0000 UTC m=+1108.658019468" Oct 31 23:08:50.905502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.905423 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:08:50.912350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.912325 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:50.917186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.917157 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-2294a7-serving-cert\"" Oct 31 23:08:50.919184 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:50.919157 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:08:51.006459 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.006425 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.006617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.006591 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdj4\" (UniqueName: \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.006694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.006644 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.006747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.006728 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.107443 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.107408 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.107647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.107453 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.107647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.107535 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.107788 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.107653 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdj4\" (UniqueName: \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.108402 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.108371 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.110190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.110167 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.110278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.110188 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.120245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.120217 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdj4\" (UniqueName: \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\") pod \"maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.228535 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.228445 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:51.507799 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.507770 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:08:51.508752 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:08:51.508728 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a39603_18f1_4383_b261_bbf664a67e61.slice/crio-534f3b9e83feb6456b1586fd02fde0c9b43f4af8764cf19780a891397a1eaf28 WatchSource:0}: Error finding container 534f3b9e83feb6456b1586fd02fde0c9b43f4af8764cf19780a891397a1eaf28: Status 404 returned error can't find the container with id 534f3b9e83feb6456b1586fd02fde0c9b43f4af8764cf19780a891397a1eaf28 Oct 31 23:08:51.658132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:51.658093 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" event={"ID":"a6a39603-18f1-4383-b261-bbf664a67e61","Type":"ContainerStarted","Data":"534f3b9e83feb6456b1586fd02fde0c9b43f4af8764cf19780a891397a1eaf28"} Oct 31 23:08:52.606156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:52.606128 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:08:52.668594 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:52.668558 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" event={"ID":"a6a39603-18f1-4383-b261-bbf664a67e61","Type":"ContainerStarted","Data":"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6"} Oct 31 23:08:52.669121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:52.668616 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:52.689383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:52.689332 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" podStartSLOduration=2.302022085 podStartE2EDuration="2.689306918s" totalImagesPullingTime="387.284833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:52.688607252 +0000 UTC m=+1110.671349631" watchObservedRunningTime="2026-10-31 23:08:52.689306918 +0000 UTC m=+1110.672049295" Oct 31 23:08:54.637490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:54.637462 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:08:55.654856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:55.654828 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:08:56.584853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:56.584823 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:08:56.585058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:56.584888 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:08:56.664834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:56.664805 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:08:58.625828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.625313 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:08:58.625828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.625808 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:08:58.626470 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.625864 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:08:58.626720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.626697 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:08:58.626853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.626763 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:08:58.626853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.626809 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:08:58.626941 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.626853 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:08:58.627041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.627022 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:08:58.680210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:58.680179 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:08:59.636828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:08:59.636795 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:09:00.652418 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:00.652389 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:09:14.888304 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:09:14.888267 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:09:22.300646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.300603 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:09:22.312489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.312457 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:09:22.312637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.312568 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.318486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.318461 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"openshift-service-ca.crt\"" Oct 31 23:09:22.320065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.320032 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"kube-root-ca.crt\"" Oct 31 23:09:22.320263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.320082 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"default-dockercfg-lql6f\"" Oct 31 23:09:22.320263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.320090 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"simulator-w6-7d7eed-kserve-self-signed-certs\"" Oct 31 23:09:22.455551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455502 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.455752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455671 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdx7\" (UniqueName: \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.455829 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455788 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.455888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455821 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.455888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455852 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.456010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.455935 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.556847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.556758 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.556866 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.556923 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdx7\" (UniqueName: \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.556999 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.557022 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.557042 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.557347 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.557441 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.557548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.557527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.559406 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.559384 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.559507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.559441 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.569612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.569585 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdx7\" (UniqueName: \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\") pod \"simulator-w6-7d7eed-kserve-649b9878c5-sd8k7\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.629174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.629147 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:22.909213 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:22.909185 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:09:22.910701 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:09:22.910675 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea39a53_3824_4c95_bb01_edf9a0383e67.slice/crio-574f2d01a4bc7d9d1fb8254bae05aac4d238a4d953e441ac01f8b43098e89197 WatchSource:0}: Error finding container 574f2d01a4bc7d9d1fb8254bae05aac4d238a4d953e441ac01f8b43098e89197: Status 404 returned error can't find the container with id 574f2d01a4bc7d9d1fb8254bae05aac4d238a4d953e441ac01f8b43098e89197 Oct 31 23:09:23.876854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:23.876818 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" event={"ID":"dea39a53-3824-4c95-bb01-edf9a0383e67","Type":"ContainerStarted","Data":"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b"} Oct 31 23:09:23.877276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:23.876865 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" event={"ID":"dea39a53-3824-4c95-bb01-edf9a0383e67","Type":"ContainerStarted","Data":"574f2d01a4bc7d9d1fb8254bae05aac4d238a4d953e441ac01f8b43098e89197"} Oct 31 23:09:23.898577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:23.898519 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podStartSLOduration=1.89850199 podStartE2EDuration="1.89850199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:23.897630631 +0000 UTC m=+1141.880373007" watchObservedRunningTime="2026-10-31 23:09:23.89850199 +0000 UTC m=+1141.881244366" Oct 31 23:09:24.629241 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:24.629203 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:32.629878 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:32.629843 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:32.642491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:32.642459 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:32.942024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:32.941930 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:09:35.409967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.409928 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:09:35.414493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.414473 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.424374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.423691 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"openshift-service-ca.crt\"" Oct 31 23:09:35.424374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.423776 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"kube-root-ca.crt\"" Oct 31 23:09:35.424374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.423789 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"premium-w2-255748-kserve-self-signed-certs\"" Oct 31 23:09:35.424374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.423867 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"default-dockercfg-5rkmz\"" Oct 31 23:09:35.425798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.425769 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:09:35.498577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498540 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.498577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498580 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdcg\" (UniqueName: \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.498794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498611 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.498830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498782 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.498830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498816 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.498895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.498847 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600270 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600310 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600448 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600578 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600604 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdcg\" (UniqueName: \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600639 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600740 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600754 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.600944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.600855 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.603121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.603098 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.603256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.603162 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.613561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.613531 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdcg\" (UniqueName: \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\") pod \"premium-w2-255748-kserve-549db6f8f5-5lndz\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:35.729451 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:35.729370 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:36.014237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:36.014205 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:09:36.962623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:36.962588 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" event={"ID":"3d48f7b7-120b-4452-9c04-121ffc7ef7ef","Type":"ContainerStarted","Data":"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a"} Oct 31 23:09:36.962623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:36.962629 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" event={"ID":"3d48f7b7-120b-4452-9c04-121ffc7ef7ef","Type":"ContainerStarted","Data":"d42d75f280ef3f09210613ab8939cb62b5a8aa4d8955709301606c66fccfc821"} Oct 31 23:09:36.985359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:36.985311 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podStartSLOduration=1.985294525 podStartE2EDuration="1.985294525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:36.983420526 +0000 UTC m=+1154.966162916" watchObservedRunningTime="2026-10-31 23:09:36.985294525 +0000 UTC m=+1154.968036901" Oct 31 23:09:37.730056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:37.730019 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:44.888827 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:09:44.888788 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:09:45.730545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:45.730507 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:45.743244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:45.743219 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:46.027797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:46.027772 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:09:48.874365 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.874333 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:09:48.879632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.879611 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.890225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.890199 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"premium-w1-04c5c2-kserve-self-signed-certs\"" Oct 31 23:09:48.890225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.890212 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"kube-root-ca.crt\"" Oct 31 23:09:48.890608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.890234 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"openshift-service-ca.crt\"" Oct 31 23:09:48.890753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.890738 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"default-dockercfg-jcdwr\"" Oct 31 23:09:48.893427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.893406 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:09:48.957820 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.957781 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.957985 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.957859 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.957985 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.957889 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.957985 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.957922 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsf8q\" (UniqueName: \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.958162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.958130 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:48.958198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:48.958163 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059649 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059702 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059756 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059823 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059853 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.059895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.059877 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsf8q\" (UniqueName: \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.060206 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.060146 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.060206 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.060168 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.060301 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.060235 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.062304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.062270 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.062653 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.062638 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.073399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.073377 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsf8q\" (UniqueName: \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\") pod \"premium-w1-04c5c2-kserve-5657c99c46-4dcvv\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.194210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.194124 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:49.474333 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:49.474301 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:09:49.477410 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:09:49.477372 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd64337a1_7f1d_4fce_ab33_79092a707849.slice/crio-7951371c91e157cc8ab7075739b8fc8759e518c4f58d7aaf00d500f3b0f3974b WatchSource:0}: Error finding container 7951371c91e157cc8ab7075739b8fc8759e518c4f58d7aaf00d500f3b0f3974b: Status 404 returned error can't find the container with id 7951371c91e157cc8ab7075739b8fc8759e518c4f58d7aaf00d500f3b0f3974b Oct 31 23:09:50.048500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:50.048450 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" event={"ID":"d64337a1-7f1d-4fce-ab33-79092a707849","Type":"ContainerStarted","Data":"02aaa7609b34818325738145e7df305823b85cb31084a8426a74dc189d70a488"} Oct 31 23:09:50.048971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:50.048516 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" event={"ID":"d64337a1-7f1d-4fce-ab33-79092a707849","Type":"ContainerStarted","Data":"7951371c91e157cc8ab7075739b8fc8759e518c4f58d7aaf00d500f3b0f3974b"} Oct 31 23:09:50.070900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:50.070842 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podStartSLOduration=2.070825062 podStartE2EDuration="2.070825062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:50.068056842 +0000 UTC m=+1168.050799223" watchObservedRunningTime="2026-10-31 23:09:50.070825062 +0000 UTC m=+1168.053567443" Oct 31 23:09:50.195141 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:50.195101 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:59.194306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:59.194270 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:09:59.206862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:09:59.206835 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:10:00.119678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:00.119647 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:10:14.887203 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:10:14.887113 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:10:22.305539 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:22.305514 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:10:22.306466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:22.306443 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:10:24.245738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.245699 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:10:24.251517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.251495 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.260679 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.260640 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a16dfb-openshift-default-dockercfg-lchgz\"" Oct 31 23:10:24.267761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.267736 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:10:24.370705 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.370675 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.370872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.370721 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.370872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.370823 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.370872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.370864 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.370971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.370950 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkkp\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.371025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.371000 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.371059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.371030 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.371103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.371090 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.371138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.371109 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472375 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472422 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472469 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472501 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472538 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkkp\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472567 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472594 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472633 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.472846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.472658 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.473038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.473010 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.473109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.473064 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.473167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.473121 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.473297 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.473279 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.473395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.473378 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.474593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.474577 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.474837 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.474819 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.490946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.490918 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.491074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.491059 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkkp\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.566139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.566106 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:24.861008 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.860906 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:10:24.865053 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:10:24.865025 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd97fdf_42aa_4d51_a2a3_65e362339cab.slice/crio-346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8 WatchSource:0}: Error finding container 346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8: Status 404 returned error can't find the container with id 346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8 Oct 31 23:10:24.867660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.867620 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:24.867767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.867712 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:24.867833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:24.867770 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:25.288520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.288462 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" event={"ID":"3cd97fdf-42aa-4d51-a2a3-65e362339cab","Type":"ContainerStarted","Data":"7c257471b7060a622c45eac4961dd4b138fe82057feed974731df1aa08bd20c1"} Oct 31 23:10:25.288973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.288574 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" event={"ID":"3cd97fdf-42aa-4d51-a2a3-65e362339cab","Type":"ContainerStarted","Data":"346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8"} Oct 31 23:10:25.316475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.316418 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podStartSLOduration=1.31640472 podStartE2EDuration="1.31640472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:25.315720044 +0000 UTC m=+1203.298462423" watchObservedRunningTime="2026-10-31 23:10:25.31640472 +0000 UTC m=+1203.299147097" Oct 31 23:10:25.573182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.571719 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:25.573182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.571885 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:25.573182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:25.571944 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:26.569534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:26.569409 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:26.569534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:26.569494 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:27.568481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:27.568440 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:27.568711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:27.568516 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:28.568371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:28.568329 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:28.568756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:28.568386 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:29.567502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:29.567462 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:29.568011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:29.567853 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:30.568544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:30.568502 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 23:10:30.569022 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:30.568573 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 23:10:31.571467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:31.571435 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:31.571900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:31.571888 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:31.572545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:31.572521 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:10:44.889152 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:10:44.889112 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:10:54.638704 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.638670 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:10:54.643321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.643293 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.670218 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.670180 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:10:54.672743 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.672710 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.672873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.672831 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.672927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.672877 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgq4\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673137 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673180 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673292 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673349 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673334 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673436 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.673526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.673502 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.705101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.705062 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-135248-openshift-default-dockercfg-g78k6\"" Oct 31 23:10:54.774531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774497 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.774745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774544 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.774745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774598 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.774745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774640 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.774745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774689 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.774745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774741 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774783 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgq4\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774881 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.774904 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.775112 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.775138 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.775393 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775539 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.775507 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.775595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.775573 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.777174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.777140 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.777299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.777284 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.786661 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.786626 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.786904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.786883 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgq4\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:54.975219 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:54.975141 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:55.270023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.269998 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:10:55.271804 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:10:55.271777 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3248837_588d_4f99_8122_5517412e2120.slice/crio-e2d6fc4ca4032d64845a5b10a20452cce16a4b0ba801b8a4abf0f6152bfa6fc7 WatchSource:0}: Error finding container e2d6fc4ca4032d64845a5b10a20452cce16a4b0ba801b8a4abf0f6152bfa6fc7: Status 404 returned error can't find the container with id e2d6fc4ca4032d64845a5b10a20452cce16a4b0ba801b8a4abf0f6152bfa6fc7 Oct 31 23:10:55.274089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.274054 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:55.274192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.274120 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:55.274192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.274148 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:10:55.508224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.508181 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" event={"ID":"a3248837-588d-4f99-8122-5517412e2120","Type":"ContainerStarted","Data":"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4"} Oct 31 23:10:55.508411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.508247 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" event={"ID":"a3248837-588d-4f99-8122-5517412e2120","Type":"ContainerStarted","Data":"e2d6fc4ca4032d64845a5b10a20452cce16a4b0ba801b8a4abf0f6152bfa6fc7"} Oct 31 23:10:55.975426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:55.975393 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:10:56.977759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:56.977724 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 23:10:56.978165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:56.977776 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 23:10:57.976400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:57.976360 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 23:10:57.976670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:57.976422 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 23:10:58.975762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:58.975727 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 23:10:58.976190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:58.975782 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 23:10:59.976456 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:59.976417 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 23:10:59.976938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:10:59.976481 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 23:11:00.981017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:00.979347 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 23:11:00.981017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:00.979419 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 23:11:02.998123 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:02.998081 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:11:03.023139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:03.008100 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.596157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:03.595537 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:11:03.596366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:03.596307 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:11:03.597563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:03.597537 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:11:03.624120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:03.624068 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podStartSLOduration=9.624054736 podStartE2EDuration="9.624054736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:55.536260747 +0000 UTC m=+1233.519003126" watchObservedRunningTime="2026-10-31 23:11:03.624054736 +0000 UTC m=+1241.606797114" Oct 31 23:11:08.863339 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.863300 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:11:08.869379 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.869342 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.877848 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.877816 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:11:08.878554 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.878530 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"distinct-w2-255748-kserve-self-signed-certs\"" Oct 31 23:11:08.907869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.907832 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.908106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.907883 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.908106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.907973 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.908246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.908109 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.908246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.908155 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qcw\" (UniqueName: \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:08.908246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:08.908221 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009233 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009289 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qcw\" (UniqueName: \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009350 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009540 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009571 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009607 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.009815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009777 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.010018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.009972 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.010249 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.010047 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.012157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.012135 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.012290 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.012274 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.022494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.022465 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qcw\" (UniqueName: \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\") pod \"distinct-w2-255748-kserve-c7449b854-ntbjk\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.172750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.172676 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:11:09.177429 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.177405 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.181320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.181298 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04c5c2-openshift-default-dockercfg-js5sc\"" Oct 31 23:11:09.187543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.187518 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:09.192929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.192902 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:11:09.212067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212027 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212123 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212149 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cgc\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212179 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212218 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212305 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212348 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212392 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.212598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.212462 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313153 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313191 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313412 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313252 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313412 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313356 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313474 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313600 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313634 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cgc\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313665 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.313870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313697 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.314174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.313870 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.314174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.314054 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.314174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.314135 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.314281 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.314212 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.314461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.314442 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.316219 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.316191 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.316311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.316225 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.326926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.326895 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.327120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.326939 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cgc\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.342347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.342311 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:11:09.347924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.347901 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.357375 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.357356 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2294a7-openshift-default-dockercfg-nh2x9\"" Oct 31 23:11:09.368967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.368938 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:11:09.414952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.414915 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.414958 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415072 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415123 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415229 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415205 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415260 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415283 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wsc\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415385 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.415448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.415435 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.442792 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.442713 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:11:09.447084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.447064 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.480683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.480642 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-255748-openshift-default-dockercfg-89xj2\"" Oct 31 23:11:09.488371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.488336 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:11:09.492972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.492948 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:09.517286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.517253 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:11:09.518348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518315 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518391 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.518478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518433 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wsc\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.518589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518550 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.518641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518609 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518699 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518668 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.518749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518724 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518752 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518775 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518805 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.518897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518849 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.518897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518876 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.519078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518928 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.519078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.518976 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.519078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519047 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.519221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519083 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.519221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519108 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwd2\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.519221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519139 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.519819 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519792 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.519933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.519829 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.520129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.520109 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.520775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.520752 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.523324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.521782 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.523324 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:09.522084 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode12de15d_7208_4703_a91d_ef282c36e0fb.slice/crio-1b397cc806b4f26ddfd6a01cf2a6c49061f394a2dc8c2f49da93d3311727ee47 WatchSource:0}: Error finding container 1b397cc806b4f26ddfd6a01cf2a6c49061f394a2dc8c2f49da93d3311727ee47: Status 404 returned error can't find the container with id 1b397cc806b4f26ddfd6a01cf2a6c49061f394a2dc8c2f49da93d3311727ee47 Oct 31 23:11:09.523324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.522546 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.527559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.527494 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.539520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.539483 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.539742 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.539698 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wsc\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.620533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620498 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620545 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620567 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620587 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620656 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620686 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwd2\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620711 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.620756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620744 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.621145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.620838 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.621505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.621485 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.621692 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.621677 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.622971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.622649 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.622971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.622713 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.622971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.622934 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.624508 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.624466 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.625763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.625740 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.640023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.639694 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.640938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.640396 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwd2\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.643813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.643786 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" event={"ID":"e12de15d-7208-4703-a91d-ef282c36e0fb","Type":"ContainerStarted","Data":"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479"} Oct 31 23:11:09.644033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.644012 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" event={"ID":"e12de15d-7208-4703-a91d-ef282c36e0fb","Type":"ContainerStarted","Data":"1b397cc806b4f26ddfd6a01cf2a6c49061f394a2dc8c2f49da93d3311727ee47"} Oct 31 23:11:09.662682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.662651 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:09.666072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.666024 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podStartSLOduration=1.666008716 podStartE2EDuration="1.666008716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:09.664771497 +0000 UTC m=+1247.647513874" watchObservedRunningTime="2026-10-31 23:11:09.666008716 +0000 UTC m=+1247.648751091" Oct 31 23:11:09.760598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.760559 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:09.804276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.804247 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:11:09.805329 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:09.805258 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6662d9f_823a_4d16_bf62_86dc28511ef7.slice/crio-c557e18d48bf0dcb0e791f0a9729dd313d809dcc88b05c38fafc135209eb5495 WatchSource:0}: Error finding container c557e18d48bf0dcb0e791f0a9729dd313d809dcc88b05c38fafc135209eb5495: Status 404 returned error can't find the container with id c557e18d48bf0dcb0e791f0a9729dd313d809dcc88b05c38fafc135209eb5495 Oct 31 23:11:09.809163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.809098 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:09.809286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.809195 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:09.809286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.809239 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:09.952153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.952120 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:11:09.953204 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:09.953113 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626a2acd_241b_46ef_a171_481512229bf3.slice/crio-d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530 WatchSource:0}: Error finding container d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530: Status 404 returned error can't find the container with id d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530 Oct 31 23:11:09.955266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.955234 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:09.955329 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.955306 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:09.955368 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:09.955337 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:10.067193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.067130 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:11:10.078457 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:10.078423 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31668b3b_cfeb_4832_a16d_3ac77c8d6694.slice/crio-963d4375e171177c6d56de9fa837b341bb6ed547298348ad3a729199b46283e7 WatchSource:0}: Error finding container 963d4375e171177c6d56de9fa837b341bb6ed547298348ad3a729199b46283e7: Status 404 returned error can't find the container with id 963d4375e171177c6d56de9fa837b341bb6ed547298348ad3a729199b46283e7 Oct 31 23:11:10.090932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.084535 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:10.093853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.092103 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:10.093853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.092240 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:10.187925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.187890 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:10.694433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.693272 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" event={"ID":"626a2acd-241b-46ef-a171-481512229bf3","Type":"ContainerStarted","Data":"e2f55d2d1896d21fab87681b86e3a8798e07a9d8d3ca6902a338dc0a04060165"} Oct 31 23:11:10.694433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.693348 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" event={"ID":"626a2acd-241b-46ef-a171-481512229bf3","Type":"ContainerStarted","Data":"d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530"} Oct 31 23:11:10.696978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.696940 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" event={"ID":"c6662d9f-823a-4d16-bf62-86dc28511ef7","Type":"ContainerStarted","Data":"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5"} Oct 31 23:11:10.699548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.699501 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" event={"ID":"c6662d9f-823a-4d16-bf62-86dc28511ef7","Type":"ContainerStarted","Data":"c557e18d48bf0dcb0e791f0a9729dd313d809dcc88b05c38fafc135209eb5495"} Oct 31 23:11:10.706230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.706197 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" event={"ID":"31668b3b-cfeb-4832-a16d-3ac77c8d6694","Type":"ContainerStarted","Data":"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4"} Oct 31 23:11:10.706599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.706577 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" event={"ID":"31668b3b-cfeb-4832-a16d-3ac77c8d6694","Type":"ContainerStarted","Data":"963d4375e171177c6d56de9fa837b341bb6ed547298348ad3a729199b46283e7"} Oct 31 23:11:10.726018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.723120 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podStartSLOduration=1.723107189 podStartE2EDuration="1.723107189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:10.722370839 +0000 UTC m=+1248.705113217" watchObservedRunningTime="2026-10-31 23:11:10.723107189 +0000 UTC m=+1248.705849567" Oct 31 23:11:10.766242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.766143 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:10.778223 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:10.778155 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podStartSLOduration=1.7781318000000002 podStartE2EDuration="1.7781318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:10.746927477 +0000 UTC m=+1248.729669850" watchObservedRunningTime="2026-10-31 23:11:10.7781318 +0000 UTC m=+1248.760874180" Oct 31 23:11:11.497012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.496355 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:11.510020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.504121 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:11.510020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.504190 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:11.668922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.668835 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:11.669254 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.669229 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:11.669413 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.669386 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:11.809075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.785438 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:11.809482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:11.809441 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:12.495014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.494431 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:12.495014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.494499 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:12.686046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.680662 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:12.686046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.680721 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:12.804048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.800266 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:12.804048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:12.800329 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:13.513017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.503248 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:13.513017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.503305 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:13.665015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.664408 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:13.665015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.664470 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:13.765250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.765152 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:13.774020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:13.765799 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:14.507015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.502424 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:14.507015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.502494 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:14.665474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.665433 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:14.665789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.665757 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:14.763892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.763790 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:14.764195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:14.764152 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:14.951019 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:11:14.944264 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:11:15.511017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.497424 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:15.511017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.497499 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:15.710035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.695103 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:15.710035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.695167 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:15.807407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.807312 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:15.807712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:15.807653 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:16.494198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.494162 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 23:11:16.494690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.494230 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 23:11:16.663095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.663056 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:16.663322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.663122 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 23:11:16.762195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.762143 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 23:11:16.762405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:16.762208 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 23:11:17.498250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.498213 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:17.498752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.498731 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:17.499552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.499523 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:11:17.524958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.524883 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podStartSLOduration=8.524863882 podStartE2EDuration="8.524863882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:10.826747701 +0000 UTC m=+1248.809490080" watchObservedRunningTime="2026-10-31 23:11:17.524863882 +0000 UTC m=+1255.507606264" Oct 31 23:11:17.668377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.668344 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:17.668749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.668722 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:17.669655 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.669636 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:11:17.766625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.766541 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:17.766957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.766937 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:17.767752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:17.767731 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:11:19.194013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:19.193393 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:19.517014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:19.515359 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:20.360641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:20.360331 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:11:24.717315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.717283 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:11:24.723689 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.723661 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.734432 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.734375 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d7eed-openshift-default-dockercfg-lp7z5\"" Oct 31 23:11:24.743880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.743843 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:11:24.784707 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.784660 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.784899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.784719 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.784899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.784819 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.784899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.784851 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.785114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.785020 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.785114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.785049 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.785114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.785095 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hxj\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.785247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.785145 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.785317 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.785292 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887065 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887127 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887190 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887224 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887328 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887356 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887381 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hxj\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887417 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887494 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887609 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887635 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.887945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887920 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.888029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.887938 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.888579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.888555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.889555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.889529 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.889845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.889816 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.900187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.900165 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hxj\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:24.900269 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:24.900214 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:25.064956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:25.064910 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:25.356591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:25.356564 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:11:25.358047 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:25.358015 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0825b221_898d_4dcc_bdef_fa0f8673a2c8.slice/crio-acc0a4a67fd71b22d55079472c9358b11dc536121a85b6bebc73cb0b9ec2b6cd WatchSource:0}: Error finding container acc0a4a67fd71b22d55079472c9358b11dc536121a85b6bebc73cb0b9ec2b6cd: Status 404 returned error can't find the container with id acc0a4a67fd71b22d55079472c9358b11dc536121a85b6bebc73cb0b9ec2b6cd Oct 31 23:11:25.360736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:25.360697 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:25.360843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:25.360776 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:25.360843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:25.360816 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:26.078016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:26.077963 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" event={"ID":"0825b221-898d-4dcc-bdef-fa0f8673a2c8","Type":"ContainerStarted","Data":"fb09f5a125f3f7a11aee172da8386956ebaf99f6b4455fea8def1fedcc60da3d"} Oct 31 23:11:26.078453 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:26.078044 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" event={"ID":"0825b221-898d-4dcc-bdef-fa0f8673a2c8","Type":"ContainerStarted","Data":"acc0a4a67fd71b22d55079472c9358b11dc536121a85b6bebc73cb0b9ec2b6cd"} Oct 31 23:11:26.106332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:26.106278 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podStartSLOduration=2.106265298 podStartE2EDuration="2.106265298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:26.104396078 +0000 UTC m=+1264.087138456" watchObservedRunningTime="2026-10-31 23:11:26.106265298 +0000 UTC m=+1264.089007720" Oct 31 23:11:27.068578 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:27.068537 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:27.074151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:27.074119 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 23:11:27.074300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:27.074182 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 23:11:28.068016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:28.067259 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 23:11:28.068016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:28.067319 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 23:11:29.065783 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:29.065743 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 23:11:29.066109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:29.066077 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 23:11:30.070387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:30.070341 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 23:11:30.070946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:30.070409 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 23:11:31.067013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:31.066269 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 23:11:31.067013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:31.066327 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 23:11:32.191901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.191867 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:32.192352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.192172 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:32.193160 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.193141 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:11:32.441422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.441389 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:11:32.450476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.450405 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:11:32.450609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.450505 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.456410 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.456382 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"distinct-w6-7d7eed-kserve-self-signed-certs\"" Oct 31 23:11:32.614374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614339 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.614588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614410 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvktl\" (UniqueName: \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.614588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614539 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.614588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614577 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.614749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614698 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.614822 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.614802 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716051 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716096 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716133 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716177 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716221 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716272 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvktl\" (UniqueName: \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716465 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716604 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.716671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.716622 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.718507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.718476 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.718655 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.718637 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.730907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.730879 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvktl\" (UniqueName: \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\") pod \"distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:32.770214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:32.770183 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:33.059129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:33.059097 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:11:33.060627 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:33.060594 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb1b6bef_79f3_4d65_98a7_e8932e6afdc1.slice/crio-6e90274b234ed9fce7283ada9fc723e7c7a845e8d7ca086fc63813da839d865a WatchSource:0}: Error finding container 6e90274b234ed9fce7283ada9fc723e7c7a845e8d7ca086fc63813da839d865a: Status 404 returned error can't find the container with id 6e90274b234ed9fce7283ada9fc723e7c7a845e8d7ca086fc63813da839d865a Oct 31 23:11:33.129809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:33.129780 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" event={"ID":"db1b6bef-79f3-4d65-98a7-e8932e6afdc1","Type":"ContainerStarted","Data":"6e90274b234ed9fce7283ada9fc723e7c7a845e8d7ca086fc63813da839d865a"} Oct 31 23:11:34.154012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:34.149007 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" event={"ID":"db1b6bef-79f3-4d65-98a7-e8932e6afdc1","Type":"ContainerStarted","Data":"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab"} Oct 31 23:11:34.213681 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:34.213609 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podStartSLOduration=2.213584683 podStartE2EDuration="2.213584683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:34.183429836 +0000 UTC m=+1272.166172214" watchObservedRunningTime="2026-10-31 23:11:34.213584683 +0000 UTC m=+1272.196327063" Oct 31 23:11:34.771019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:34.770969 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:39.608943 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.608908 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:11:39.614082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.614055 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.622047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.621980 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-67fce9-openshift-default-dockercfg-vsgs4\"" Oct 31 23:11:39.630138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.630105 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643386 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jbs\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643448 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643517 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643557 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643648 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643670 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643713 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643804 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.645038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.643859 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.719536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.719499 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:11:39.724449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.724425 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.740123 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.740077 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:11:39.747035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.746979 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.747334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747312 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.747478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747461 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.747664 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747634 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.747774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747671 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.747774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747723 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jbs\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.747774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747758 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9dj\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747789 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747849 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747869 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747887 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747960 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.747976 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748047 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748073 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748139 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.748609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748169 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.748609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748194 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.748609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.748549 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.749547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.749521 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.749749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.749732 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.749972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.749956 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.750886 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.750806 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.753378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.753313 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.753731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.753706 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.778052 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.778018 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.778688 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.778665 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jbs\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:39.849561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849525 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.849561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849570 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.849804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849589 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.849859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849806 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.849916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849902 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.849966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.849933 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850052 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850006 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9dj\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850097 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850104 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850154 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850278 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850362 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850313 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.850484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850465 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.851023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.850960 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.852720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.852651 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.852837 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.852747 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.862715 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.862645 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.862852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.862746 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9dj\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:39.931963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:39.931913 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:40.055268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.055231 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:40.222618 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.222589 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:11:40.224703 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:40.224673 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfea965_40cb_411b_9aa4_4ca947983ed2.slice/crio-964976912a993a08c3afa6cb6fe1571c17de41400a5ff90e4b3fe0ff27a4ad56 WatchSource:0}: Error finding container 964976912a993a08c3afa6cb6fe1571c17de41400a5ff90e4b3fe0ff27a4ad56: Status 404 returned error can't find the container with id 964976912a993a08c3afa6cb6fe1571c17de41400a5ff90e4b3fe0ff27a4ad56 Oct 31 23:11:40.227850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.227811 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:40.227957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.227896 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:40.227957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.227942 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:40.353434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.353403 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:11:40.355196 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:40.355160 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c7f385_bc1e_43a1_8f3a_b0d7a44dba85.slice/crio-cbc2cabb3bd722effeab4bee19d599ded0e8cdba7b7b128115954187fdf2b1b9 WatchSource:0}: Error finding container cbc2cabb3bd722effeab4bee19d599ded0e8cdba7b7b128115954187fdf2b1b9: Status 404 returned error can't find the container with id cbc2cabb3bd722effeab4bee19d599ded0e8cdba7b7b128115954187fdf2b1b9 Oct 31 23:11:40.358841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.358804 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:40.358929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.358887 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:40.358969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:40.358928 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:11:41.215769 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.215728 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" event={"ID":"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85","Type":"ContainerStarted","Data":"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4"} Oct 31 23:11:41.216298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.216114 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" event={"ID":"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85","Type":"ContainerStarted","Data":"cbc2cabb3bd722effeab4bee19d599ded0e8cdba7b7b128115954187fdf2b1b9"} Oct 31 23:11:41.217774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.217745 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" event={"ID":"ccfea965-40cb-411b-9aa4-4ca947983ed2","Type":"ContainerStarted","Data":"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2"} Oct 31 23:11:41.217948 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.217785 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" event={"ID":"ccfea965-40cb-411b-9aa4-4ca947983ed2","Type":"ContainerStarted","Data":"964976912a993a08c3afa6cb6fe1571c17de41400a5ff90e4b3fe0ff27a4ad56"} Oct 31 23:11:41.245684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.243306 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podStartSLOduration=2.2432874959999998 podStartE2EDuration="2.243287496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.24040526 +0000 UTC m=+1279.223147671" watchObservedRunningTime="2026-10-31 23:11:41.243287496 +0000 UTC m=+1279.226029875" Oct 31 23:11:41.296021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.295198 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podStartSLOduration=2.295175758 podStartE2EDuration="2.295175758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.290257788 +0000 UTC m=+1279.273000170" watchObservedRunningTime="2026-10-31 23:11:41.295175758 +0000 UTC m=+1279.277918136" Oct 31 23:11:41.932393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.932366 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:41.933889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.933866 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:41.934010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:41.933912 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:42.065440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.063593 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:42.065440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.065116 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:42.065440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.065171 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:42.781015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.775337 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:42.807321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.807293 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:42.932850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.932817 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:42.933047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:42.932876 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:43.071065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:43.059036 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:43.071065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:43.059093 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:43.242519 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:43.242488 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:11:43.932746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:43.932703 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:43.933226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:43.932764 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:44.066021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:44.062978 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:44.066021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:44.063056 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:44.917698 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:11:44.917657 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:11:44.932528 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:44.932498 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:44.932682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:44.932545 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:45.064017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:45.061401 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:45.064017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:45.061465 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:45.932782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:45.932742 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:45.932977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:45.932794 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:46.064015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:46.059394 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:46.064015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:46.059455 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:46.932884 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:46.932843 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 23:11:46.933115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:46.932923 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 23:11:47.055866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:47.055832 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 23:11:47.056259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:47.055894 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 23:11:47.936591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:47.936559 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:47.937004 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:47.936951 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:47.937606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:47.937589 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:11:48.060605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.060574 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:48.061051 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.061024 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:48.061701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.061683 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:11:48.949660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.949629 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:11:48.955767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.955742 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:48.963088 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.963042 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-255748\"/\"distinct-2-w2-255748-kserve-self-signed-certs\"" Oct 31 23:11:48.967499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:48.966965 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:11:49.044685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.044651 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.044685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.044692 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.044900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.044797 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.044900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.044826 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.044900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.044876 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.045048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.045028 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hx2c\" (UniqueName: \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146442 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146651 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146505 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146651 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146557 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hx2c\" (UniqueName: \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146651 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146647 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146673 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146710 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146859 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.146943 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.146915 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.147131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.147111 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.148787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.148765 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.149205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.149189 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.166356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.166295 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hx2c\" (UniqueName: \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\") pod \"distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.276792 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.276754 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:49.562142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:49.562113 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:11:49.563968 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:49.563933 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd45f0f22_b29b_470e_8e69_57e1298605e6.slice/crio-d060ad9dc8e41d6fafac54c1998f10fc88d02cbb92811e541951d506ec8fa384 WatchSource:0}: Error finding container d060ad9dc8e41d6fafac54c1998f10fc88d02cbb92811e541951d506ec8fa384: Status 404 returned error can't find the container with id d060ad9dc8e41d6fafac54c1998f10fc88d02cbb92811e541951d506ec8fa384 Oct 31 23:11:50.335026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:50.334976 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" event={"ID":"d45f0f22-b29b-470e-8e69-57e1298605e6","Type":"ContainerStarted","Data":"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed"} Oct 31 23:11:50.335026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:50.335033 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" event={"ID":"d45f0f22-b29b-470e-8e69-57e1298605e6","Type":"ContainerStarted","Data":"d060ad9dc8e41d6fafac54c1998f10fc88d02cbb92811e541951d506ec8fa384"} Oct 31 23:11:50.361907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:50.361856 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podStartSLOduration=2.361841177 podStartE2EDuration="2.361841177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:50.360314607 +0000 UTC m=+1288.343056986" watchObservedRunningTime="2026-10-31 23:11:50.361841177 +0000 UTC m=+1288.344583555" Oct 31 23:11:51.277586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:51.277550 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:56.981433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.981402 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6"] Oct 31 23:11:56.987265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.987237 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:56.991396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.991369 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 23:11:56.991396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.991386 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 23:11:56.991553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.991387 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v7rzj\"" Oct 31 23:11:56.991553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.991398 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 23:11:56.999845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:56.999813 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6"] Oct 31 23:11:57.161333 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161295 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161333 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161333 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161489 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9m4\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-kube-api-access-bb9m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161517 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161693 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161732 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c436d9bc-d717-4c14-8eac-ea035d73abbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.161798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.161785 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.262966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.262933 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.262979 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c436d9bc-d717-4c14-8eac-ea035d73abbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263048 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263122 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263138 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263364 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263248 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9m4\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-kube-api-access-bb9m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263364 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263277 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.263972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.263939 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.265512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.265486 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c436d9bc-d717-4c14-8eac-ea035d73abbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.265712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.265691 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.265805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.265789 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.265930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.265910 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.275587 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.275563 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.275696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.275678 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9m4\" (UniqueName: \"kubernetes.io/projected/c436d9bc-d717-4c14-8eac-ea035d73abbe-kube-api-access-bb9m4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j82k6\" (UID: \"c436d9bc-d717-4c14-8eac-ea035d73abbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.306875 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.306851 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:11:57.595965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.595936 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6"] Oct 31 23:11:57.597196 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:11:57.597163 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc436d9bc_d717_4c14_8eac_ea035d73abbe.slice/crio-3d0943daadb41842306395a42b56eb558699eac3a6c10bdeb106fd847513c343 WatchSource:0}: Error finding container 3d0943daadb41842306395a42b56eb558699eac3a6c10bdeb106fd847513c343: Status 404 returned error can't find the container with id 3d0943daadb41842306395a42b56eb558699eac3a6c10bdeb106fd847513c343 Oct 31 23:11:57.610602 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:57.610571 7074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:11:58.403584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:58.403545 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" event={"ID":"c436d9bc-d717-4c14-8eac-ea035d73abbe","Type":"ContainerStarted","Data":"3d0943daadb41842306395a42b56eb558699eac3a6c10bdeb106fd847513c343"} Oct 31 23:11:59.278028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:59.277283 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:59.306015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:59.305433 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:11:59.433053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:11:59.431773 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:12:02.613067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:02.613028 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:02.613343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:02.613125 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:03.492027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:03.491958 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" event={"ID":"c436d9bc-d717-4c14-8eac-ea035d73abbe","Type":"ContainerStarted","Data":"4cea7384e145d3a2b6bea11100b6582c643c6df0db72a3a17bb392d3cd9d11b8"} Oct 31 23:12:03.492223 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:03.492042 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:12:03.494403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:03.494376 7074 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j82k6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:12:03.494516 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:03.494431 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" podUID="c436d9bc-d717-4c14-8eac-ea035d73abbe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:12:03.522869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:03.522792 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" podStartSLOduration=2.5219089 podStartE2EDuration="7.522771038s" totalImagesPullingTime="5.000862138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:03.519253323 +0000 UTC m=+1301.501995712" watchObservedRunningTime="2026-10-31 23:12:03.522771038 +0000 UTC m=+1301.505513420" Oct 31 23:12:04.500709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:04.500682 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j82k6" Oct 31 23:12:09.298931 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.298899 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:12:09.304289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.304265 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.326071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.325960 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:12:09.360764 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.360724 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.360972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.360797 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.360972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.360852 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.360972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.360887 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvffh\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.360972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.360926 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.361258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.361051 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.361258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.361078 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.361258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.361171 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.361258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.361218 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.450482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.450442 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:12:09.455614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.455578 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.462163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462129 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462205 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462258 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462291 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvffh\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462605 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462668 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462782 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.462866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462819 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.463080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.462898 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.463462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.463433 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.463579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.463501 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.463923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.463864 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.464469 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.464282 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.464577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.464527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.465615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.465595 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.466708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.466685 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.472528 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.472500 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:12:09.486169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.485233 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.487610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.487579 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvffh\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.565153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565051 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565094 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnptj\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565286 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565319 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565451 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565491 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565512 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565565 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.565638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.565626 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.619741 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.619704 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:09.667346 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667313 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667346 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667359 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667383 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667405 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667457 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667512 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667528 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnptj\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667623 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.667712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.667641 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.668492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.668149 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.668492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.668192 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.668492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.668452 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.668772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.668751 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.668956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.668938 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.669945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.669926 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.670231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.670210 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.680459 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.680432 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnptj\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.681392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.681372 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.822494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.822396 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:09.914617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.914585 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:12:09.914731 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:12:09.914684 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac96af3_4543_4010_a8c3_616ac77264da.slice/crio-e84f0044341866064dea4ebb1caecf7071c6c4a57f4ca2ca356051a14d8cff6d WatchSource:0}: Error finding container e84f0044341866064dea4ebb1caecf7071c6c4a57f4ca2ca356051a14d8cff6d: Status 404 returned error can't find the container with id e84f0044341866064dea4ebb1caecf7071c6c4a57f4ca2ca356051a14d8cff6d Oct 31 23:12:09.918403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.918356 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:09.918522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.918452 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:09.918583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:09.918521 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:10.112197 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:12:10.112160 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1178048_56ee_4374_a2ed_a9478f96705c.slice/crio-21bc22c129fdc035b90d079ecc1086bf5c2f147877cb08c83f06fb434644a163 WatchSource:0}: Error finding container 21bc22c129fdc035b90d079ecc1086bf5c2f147877cb08c83f06fb434644a163: Status 404 returned error can't find the container with id 21bc22c129fdc035b90d079ecc1086bf5c2f147877cb08c83f06fb434644a163 Oct 31 23:12:10.112350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.112323 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:12:10.115494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.115404 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:10.115610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.115548 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:10.115610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.115592 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:10.567507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.567465 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" event={"ID":"2ac96af3-4543-4010-a8c3-616ac77264da","Type":"ContainerStarted","Data":"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359"} Oct 31 23:12:10.568268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.568240 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" event={"ID":"2ac96af3-4543-4010-a8c3-616ac77264da","Type":"ContainerStarted","Data":"e84f0044341866064dea4ebb1caecf7071c6c4a57f4ca2ca356051a14d8cff6d"} Oct 31 23:12:10.580256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.580206 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" event={"ID":"f1178048-56ee-4374-a2ed-a9478f96705c","Type":"ContainerStarted","Data":"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06"} Oct 31 23:12:10.580457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.580294 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" event={"ID":"f1178048-56ee-4374-a2ed-a9478f96705c","Type":"ContainerStarted","Data":"21bc22c129fdc035b90d079ecc1086bf5c2f147877cb08c83f06fb434644a163"} Oct 31 23:12:10.600936 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.600875 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podStartSLOduration=1.6008588540000002 podStartE2EDuration="1.600858854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:10.598160115 +0000 UTC m=+1308.580902494" watchObservedRunningTime="2026-10-31 23:12:10.600858854 +0000 UTC m=+1308.583601232" Oct 31 23:12:10.620920 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.620868 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:10.633951 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.633902 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podStartSLOduration=1.633889044 podStartE2EDuration="1.633889044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:10.630761702 +0000 UTC m=+1308.613504081" watchObservedRunningTime="2026-10-31 23:12:10.633889044 +0000 UTC m=+1308.616631421" Oct 31 23:12:10.827408 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:10.827200 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:11.622086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:11.622054 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:11.622479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:11.622107 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:11.824541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:11.824504 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:11.824701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:11.824564 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:12.620648 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:12.620618 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:12.620814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:12.620666 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:12.823060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:12.823029 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:12.823423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:12.823087 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:13.621424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:13.621209 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:13.621631 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:13.621474 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:13.823772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:13.823726 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:13.824198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:13.823801 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:14.620718 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:14.620647 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:14.620914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:14.620768 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:14.823548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:14.823508 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:14.823716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:14.823576 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:14.916256 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:12:14.916174 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:12:15.620308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:15.620271 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:15.620524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:15.620349 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:15.823058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:15.823021 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:15.823214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:15.823075 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:16.620420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:16.620370 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:12:16.620869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:16.620467 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 23:12:16.822853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:16.822814 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:12:16.823072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:16.822884 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 23:12:17.705691 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.705655 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:17.706353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.706328 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:17.707163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.707141 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:12:17.829150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.829120 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:17.829584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.829496 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:17.830910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:17.830883 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:12:24.451121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.451088 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:12:24.457543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.457511 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.497414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.490573 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:12:24.522481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.522447 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:12:24.528489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.528464 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.542034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.542005 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607533 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-629ql\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607660 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607689 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607717 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607761 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607822 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.607976 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.608033 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.611862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.608061 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.621911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.621878 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:12:24.627571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.627538 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:12:24.627756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.627662 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.708935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.708849 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.708935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.708890 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.708935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.708917 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.708955 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.708983 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709035 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709090 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709144 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709221 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709252 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709279 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709341 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709393 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709441 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709534 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709473 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709826 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709578 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hzs\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.709826 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709609 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629ql\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.709826 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.709691 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.710520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.710496 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.711044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.710981 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.711197 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.711179 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.711313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.711294 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.711416 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.711399 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.711821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.711779 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.723468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.723442 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.731637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.731555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.732886 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.732864 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629ql\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.770320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.770288 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:12:24.777059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.777033 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:24.790308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.790278 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:12:24.816394 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.816368 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:24.825339 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.825306 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.825566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.825541 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.825695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.825679 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.825846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.825830 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.825963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.825947 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.826116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826095 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.826344 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826326 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.826479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826462 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdqn\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.826634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826616 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hzs\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.826761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826745 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.826907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.826889 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.827050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827032 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.827188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827171 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.827386 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827369 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.827510 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827495 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.827639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827621 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.827780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827764 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:24.827968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.827950 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.831292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.828210 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.831292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.828906 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.831292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.828973 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.831292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.829208 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.831292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.829547 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.833571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.833525 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.847012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.843554 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.887535 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.887499 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hzs\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.887873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.887838 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:12:24.966905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.961010 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:24.978306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.978280 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:12:24.978606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.978591 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.003036 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:24.997512 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:12:25.028754 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.028719 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2676f\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.029016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.028981 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdqn\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.029166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029150 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.029301 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029288 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.029403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029388 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.029516 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029503 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.029672 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029659 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.029799 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029780 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.030078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.030053 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.030263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.030236 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.030431 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.029723 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.030518 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.030415 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.039254 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.039228 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.039628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.039602 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.039834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.039818 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.039961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.039939 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.040794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.040770 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.048745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.048720 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.053115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.053092 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.053302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.053286 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.053620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.046370 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.053731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.031781 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.053814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.042393 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.053925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.032356 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.054953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.035614 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.067161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.067060 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:12:25.085263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.083690 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.085263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.084196 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.090062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.090029 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdqn\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-nftll\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.191045 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.278631 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.278699 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280185 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280267 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280348 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280430 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280511 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280543 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280582 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280609 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.283600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280669 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8xr\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.280710 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2676f\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.283876 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284036 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284111 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284162 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284198 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284231 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284269 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284392 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrxf\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284442 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284502 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284531 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284558 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284595 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284657 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.284916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.284738 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.286116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.285972 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.286307 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.286568 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.286881 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.292754 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.301625 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.305560 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.308198 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.329443 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.353044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.345176 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2676f\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.386456 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386416 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.386625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386466 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.386625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386492 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.386625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386531 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.386625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386577 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.386839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386639 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.386839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386771 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.386839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386834 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386861 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386911 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8xr\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.386958 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387017 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387058 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387095 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387124 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387149 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387241 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrxf\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.387457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.387273 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.388332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.388096 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.389041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.388959 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.389855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.389808 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.393809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.393055 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.393809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.393538 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.393967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.393953 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.394479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.394457 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.395065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.395044 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.395450 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.395425 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.395833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.395814 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.396944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.396922 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.398644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.398606 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.402735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.401527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.403245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.403226 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.406201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.406181 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:25.416301 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.416278 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrxf\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.418045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.416932 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8xr\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.418849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.418807 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.421071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.421029 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.434014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.432465 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:25.576314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.575870 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:25.644355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.641744 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:12:25.887030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.886911 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:25.887143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.887077 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:25.887143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.887132 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:25.995104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:25.995062 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" event={"ID":"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9","Type":"ContainerStarted","Data":"a1294225b29a79ce11f7faf879b66ca73beb88b353d284c9896c728942b2914c"} Oct 31 23:12:26.577238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.577188 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:12:26.657030 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:12:26.625328 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc193e2ae_7f72_4030_9cf9_d930c8c8255a.slice/crio-d7b0aa1e1bc6d443237169db4fd8a5c1449e1a9734b98738150e4fb49c0cc078 WatchSource:0}: Error finding container d7b0aa1e1bc6d443237169db4fd8a5c1449e1a9734b98738150e4fb49c0cc078: Status 404 returned error can't find the container with id d7b0aa1e1bc6d443237169db4fd8a5c1449e1a9734b98738150e4fb49c0cc078 Oct 31 23:12:26.657030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.655869 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.714914 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.715022 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.715067 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.718343 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.718422 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.722719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.718468 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.731454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.731337 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:12:26.767334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.753877 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:12:26.767334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.764542 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.767334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.764629 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.767334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.764671 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.806025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.792843 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:12:26.806025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.795548 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.806025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.795662 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:26.806025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:26.795722 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:27.035061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.035021 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" event={"ID":"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9","Type":"ContainerStarted","Data":"af5931a38b8cb72f95d67dd5f2e7ad2cf0f5a2908b6cb311fde9eebdcd5aa37a"} Oct 31 23:12:27.055225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.042018 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:27.055225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.042114 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:27.055225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.042160 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:27.055225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.049554 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" event={"ID":"384aa41f-ca9b-4287-b210-856ebe1eab8a","Type":"ContainerStarted","Data":"d7be599194f0e2658859fe475026f8da6b6fa657ee5fd73d6b7d30a909c8dd3d"} Oct 31 23:12:27.055225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.053877 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" event={"ID":"c193e2ae-7f72-4030-9cf9-d930c8c8255a","Type":"ContainerStarted","Data":"d7b0aa1e1bc6d443237169db4fd8a5c1449e1a9734b98738150e4fb49c0cc078"} Oct 31 23:12:27.087185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.087153 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" event={"ID":"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9","Type":"ContainerStarted","Data":"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf"} Oct 31 23:12:27.154821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.154786 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" event={"ID":"c139df47-d3de-4818-805f-05810cdccb12","Type":"ContainerStarted","Data":"a1d066c669e44a25307e3cba470ac8d03957018420f9271174ffa16681562505"} Oct 31 23:12:27.172312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.172231 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" event={"ID":"15558e00-bb89-4cac-9cd4-866a5f89d589","Type":"ContainerStarted","Data":"34489b740629b36a435efbc362f0f4ae989817efc16683bbfd8fe58efbdf3262"} Oct 31 23:12:27.186928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.186870 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podStartSLOduration=3.186847928 podStartE2EDuration="3.186847928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:27.155493464 +0000 UTC m=+1325.138235842" watchObservedRunningTime="2026-10-31 23:12:27.186847928 +0000 UTC m=+1325.169590306" Oct 31 23:12:27.836032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.835359 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:27.836032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.835489 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:27.836032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:27.835530 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:28.505959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.501019 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" event={"ID":"c193e2ae-7f72-4030-9cf9-d930c8c8255a","Type":"ContainerStarted","Data":"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c"} Oct 31 23:12:28.529084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.528687 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" event={"ID":"c139df47-d3de-4818-805f-05810cdccb12","Type":"ContainerStarted","Data":"ebdcda9c57981b856ba6fba3aee2402a9f21af8f47657b455d3220414aed01b1"} Oct 31 23:12:28.574044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.570287 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podStartSLOduration=4.570264627 podStartE2EDuration="4.570264627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.536841959 +0000 UTC m=+1326.519584335" watchObservedRunningTime="2026-10-31 23:12:28.570264627 +0000 UTC m=+1326.553007005" Oct 31 23:12:28.584015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.582304 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" event={"ID":"15558e00-bb89-4cac-9cd4-866a5f89d589","Type":"ContainerStarted","Data":"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975"} Oct 31 23:12:28.622016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.608238 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podStartSLOduration=4.608207226 podStartE2EDuration="4.608207226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.606094565 +0000 UTC m=+1326.588836978" watchObservedRunningTime="2026-10-31 23:12:28.608207226 +0000 UTC m=+1326.590949606" Oct 31 23:12:28.622016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.609557 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" event={"ID":"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9","Type":"ContainerStarted","Data":"abb45ee047311dbdf4c2fd5468750e93bff6789370bee851d91a9d88947b0fb6"} Oct 31 23:12:28.622358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.622325 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" event={"ID":"384aa41f-ca9b-4287-b210-856ebe1eab8a","Type":"ContainerStarted","Data":"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5"} Oct 31 23:12:28.646813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.645568 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podStartSLOduration=4.64552954 podStartE2EDuration="4.64552954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.641330386 +0000 UTC m=+1326.624072757" watchObservedRunningTime="2026-10-31 23:12:28.64552954 +0000 UTC m=+1326.628271919" Oct 31 23:12:28.726673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.726444 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podStartSLOduration=4.726419265 podStartE2EDuration="4.726419265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.670744453 +0000 UTC m=+1326.653486832" watchObservedRunningTime="2026-10-31 23:12:28.726419265 +0000 UTC m=+1326.709161644" Oct 31 23:12:28.748038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.741697 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podStartSLOduration=4.741676701 podStartE2EDuration="4.741676701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:28.708521786 +0000 UTC m=+1326.691264160" watchObservedRunningTime="2026-10-31 23:12:28.741676701 +0000 UTC m=+1326.724419080" Oct 31 23:12:28.952717 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:28.952623 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.061038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.060961 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:29.281325 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.281267 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.282537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.281721 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:29.283767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.283750 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:29.325884 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.325853 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:29.326725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.326696 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.326915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.326892 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:29.414677 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.414621 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:29.439015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.418093 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.439015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.418172 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:29.440106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.440055 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:29.535070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.526564 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.535070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.526630 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:29.618012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.601568 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:29.755016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.753762 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.755016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.753824 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:29.858286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.858172 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.858564 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:29.858529 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:30.011169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.006098 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.071416 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.065081 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.091205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.084008 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.113336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.113236 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.191399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.191361 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.191649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.191622 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:30.211042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.210959 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 23:12:30.250118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.250066 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 31 23:12:30.273015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.272308 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.317011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.311129 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.317011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.311195 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:30.334023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.333743 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.412071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.411418 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.412071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.411480 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:30.432014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.417093 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:30.475015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.474125 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.475015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.474194 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:30.595535 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.595497 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.595789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.595762 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:30.830014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.829430 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.830014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.829499 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:30.942568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:30.940575 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:12:31.057182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.057130 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.073050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.058142 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:12:31.073050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.071863 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.073050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.072406 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:12:31.102999 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.100542 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.102999 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.101792 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:12:31.123515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.118185 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.147014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.145878 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:12:31.244054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.243205 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.244054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.243266 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:31.310041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.275389 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.310041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.275788 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:12:31.358164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.342627 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.358164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.342686 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:31.454032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.422088 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:12:31.454032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.422729 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:12:31.454032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.451790 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.454032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.451843 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:31.584967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.584931 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.585268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.585236 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:31.620240 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.612696 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.620240 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.612767 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:31.853014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.851590 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.853014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.851658 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:31.942071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.941266 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:31.942071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:31.941913 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:12:32.089015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.059207 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.091122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.091075 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.125011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.112798 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.127238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.125607 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:12:32.232068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.230013 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.232068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.230070 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:32.304023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.302088 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.362014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.361146 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.362014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.361205 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:32.422733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.422682 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.423358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.423329 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.423596 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.423552 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:32.466605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.466566 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.466869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.466842 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:32.583177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.583141 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.583475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.583439 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:32.834945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.834846 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.835291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.835258 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:32.961016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:32.954577 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:12:33.086069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.085542 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:12:33.096300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.094451 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:33.146231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.146178 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:33.150011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.146570 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:12:33.199014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.194485 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.199014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.194548 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:33.310153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.310091 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:33.314014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.311569 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.314014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.311625 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:33.415072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.411402 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.415072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.411466 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:33.426282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.426227 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:12:33.451025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.445356 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.451299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.451262 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:33.592521 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.592486 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.592821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.592779 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:33.593480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.593453 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded" Oct 31 23:12:33.654216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.654104 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:33.774523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.774184 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:33.946260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.946223 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.946536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.946509 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:33.969012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:33.956071 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.087195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.087092 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:12:34.100749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.100704 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.151102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.151057 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.151699 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.151667 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:12:34.207274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.207228 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.207574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.207525 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:34.307426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.307383 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.307706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.307677 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:34.313108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.313068 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.410898 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.410379 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.410898 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.410442 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:34.428844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.428069 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.437684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.437652 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.437944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.437917 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:34.580527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.580480 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.580826 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.580784 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:34.829470 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.829434 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.829783 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:34.829729 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:35.086099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.073893 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:12:35.108149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.108092 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.137726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.137403 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.172080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.171169 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.172080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.171911 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.267015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.191819 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.267015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.191881 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:35.337072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.331418 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:12:35.395021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.337722 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.447424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.447387 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:35.521114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.433196 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.538012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.431358 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.538012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.528436 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:35.538012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.443884 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.538012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.528495 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:35.638344 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.638240 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.638635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.638584 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:35.841167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.841131 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.878788 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:35.878742 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 23:12:36.118017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.105690 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.161624 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.161572 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.162279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.162244 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:12:36.221788 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.197078 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.221788 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.201625 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:12:36.296013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.266522 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.296013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.266585 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:36.398244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.398095 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:12:36.579236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.469672 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.579515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.579479 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:36.579686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.494016 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.579800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.579781 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:36.591110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.591078 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.591795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.591769 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:36.591977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.591331 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.624062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.592146 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.624062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.592231 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:36.819180 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:36.819132 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 31 23:12:37.132367 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.132269 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:37.168274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.168220 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:37.168901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.168872 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:12:37.209156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.209105 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:37.212149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.209395 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:37.240263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.240215 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.240569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.240526 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:37.307906 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.307442 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.307906 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.307500 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:37.408082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.407299 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.408082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.407362 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:37.439242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.437935 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.439242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.438028 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:37.585046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.578600 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:37.585046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.580028 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.585046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.580072 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:37.607270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.606108 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:12:37.743532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.741407 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:37.743532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.743134 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:37.777161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:37.765303 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:12:38.257898 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.257853 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:38.258936 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.258891 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:12:38.372551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.337161 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:38.372850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.372805 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:38.399025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.339824 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:12:38.399601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.341298 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:38.399807 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.399780 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:38.399927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.339172 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:12:38.460014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.448812 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:38.467671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.467632 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:38.467926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.467894 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:38.496880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.496847 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:38.497137 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.497109 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:38.601066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.594796 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:12:38.601066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.595868 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:38.601066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.595916 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:38.614034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:38.610135 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:12:39.199893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.196673 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 23:12:39.199893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.196733 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 23:12:39.259329 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.259282 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:39.312012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.310031 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:39.312012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.310096 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:39.374158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.373065 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:39.408072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.408030 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 31 23:12:39.408366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.408331 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 31 23:12:39.426013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.419470 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:12:39.489078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.426809 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:12:39.489078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.459096 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:39.489078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.459149 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:39.489078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.465145 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:39.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.545465 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:12:39.575797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.575769 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:12:39.578765 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.578737 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:39.579024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.578970 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:39.588042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.587002 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:12:39.653972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.653930 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:39.663711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.662248 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:12:39.765308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.761978 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:12:39.765308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.762054 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:12:39.891312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.891262 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:12:39.920772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.920716 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:12:39.920772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.920762 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:12:39.921093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:39.921073 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131369 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131431 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131711 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131766 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131794 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131840 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131903 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.131940 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kfn\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.133683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.132002 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.195821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.195788 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:40.233756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233714 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.233965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233780 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.233965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233808 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.233965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233867 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.233965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233928 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.234267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.233965 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kfn\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.234267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.234035 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.234267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.234202 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.234267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.234241 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.235422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.235391 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.235702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.235676 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.236349 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.236322 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.237072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.237050 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.238496 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.238474 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.243628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.239487 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.256593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.256571 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.272741 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.272711 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kfn\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.286009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.284385 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.304676 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.303590 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:40.304676 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.303960 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:12:40.304676 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.304064 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 23:12:40.328444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.326830 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:40.426371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.426337 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:40.458155 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.458115 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:40.469787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.460767 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 23:12:40.476904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.474380 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 23:12:40.578252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.577678 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 23:12:40.578252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.577735 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 23:12:40.578252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:40.578228 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:41.063195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.063152 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:41.094800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.094763 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:12:41.107448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.107255 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:12:41.195499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.195452 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:41.195804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.195776 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:41.195971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.195950 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:41.205831 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.205230 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:41.478111 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.477027 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" event={"ID":"97a5197a-1b71-42ea-ae3d-705d0396704a","Type":"ContainerStarted","Data":"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1"} Oct 31 23:12:41.478111 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.477083 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" event={"ID":"97a5197a-1b71-42ea-ae3d-705d0396704a","Type":"ContainerStarted","Data":"46e1c3aaacd4ecc4772172c72d5c0fa7aea6823af425307e203bc97619d42785"} Oct 31 23:12:41.480484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.480458 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:12:41.507310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.507250 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podStartSLOduration=2.507229936 podStartE2EDuration="2.507229936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:41.502466066 +0000 UTC m=+1339.485208445" watchObservedRunningTime="2026-10-31 23:12:41.507229936 +0000 UTC m=+1339.489972316" Oct 31 23:12:41.671904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.671819 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:41.673905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.673881 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:41.678080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.678060 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:12:41.678392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.678375 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:12:41.757950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.757913 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:41.766158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:41.766124 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:12:42.328156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:42.328128 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:42.329962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:42.329934 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:42.330091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:42.330003 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:43.327573 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:43.327537 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:43.327966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:43.327592 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:44.328334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:44.328301 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:44.328807 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:44.328350 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:44.896747 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:12:44.896711 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:12:45.332646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:45.332571 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:45.333157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:45.332693 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:46.327627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:46.327581 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:46.327849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:46.327653 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:47.327454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:47.327423 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:47.327937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:47.327901 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:48.328314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:48.328270 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:12:48.328779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:48.328343 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 23:12:49.332187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:49.332159 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:49.332674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:49.332657 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:49.333809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:49.333785 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:12:54.939180 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:54.939136 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:12:55.070013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.066195 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.115023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.100211 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.141250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.141201 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.154235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.154182 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.194574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.124262 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.194907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.194877 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195067 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8nvp\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195204 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195367 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195351 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195473 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195612 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.195821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.195806 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.196099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.196074 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.274132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.274089 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.312178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.312139 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.332151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.312547 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.333326 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.333294 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.333498 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.333477 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8nvp\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.333633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.333617 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.333769 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.333754 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.333894 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.333878 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.349591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.319114 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.349591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.317641 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.349591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.348351 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.349591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.348970 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.350370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.350351 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.384548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.354256 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.385175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.385058 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.386179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.354937 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.415005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.414683 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8nvp\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.434736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.434337 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:12:55.450088 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.441925 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.476178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.476142 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:12:55.476927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.476904 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.518579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.518509 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:55.743319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:55.743230 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:56.146014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.145349 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.190016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.148909 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:12:56.259056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.221329 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:12:56.259056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.232359 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:12:56.314068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.310311 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.357579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.357472 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.610871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:56.610771 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:57.048567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.045644 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 31 23:12:57.303154 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:12:57.303108 7074 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.096s" Oct 31 23:12:57.303759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.303740 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:12:57.303896 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.303885 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:12:57.304079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304068 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:12:57.304256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304246 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:12:57.304360 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304351 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:12:57.304461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304453 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:12:57.304552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304544 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:12:57.304644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.304636 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:12:57.363572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.363163 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:12:57.536654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.536629 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:12:57.537731 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:12:57.537704 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9a0210_fc3a_4c07_a5b0_1b15f2acc656.slice/crio-7f667bca506938655fe0f7a35b8833ba69807704b5d53758922010ddefef4187 WatchSource:0}: Error finding container 7f667bca506938655fe0f7a35b8833ba69807704b5d53758922010ddefef4187: Status 404 returned error can't find the container with id 7f667bca506938655fe0f7a35b8833ba69807704b5d53758922010ddefef4187 Oct 31 23:12:57.540945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.540912 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:57.541068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.540974 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:57.541068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:57.541042 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:12:58.338450 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.338404 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" event={"ID":"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656","Type":"ContainerStarted","Data":"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253"} Oct 31 23:12:58.338957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.338464 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" event={"ID":"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656","Type":"ContainerStarted","Data":"7f667bca506938655fe0f7a35b8833ba69807704b5d53758922010ddefef4187"} Oct 31 23:12:58.376778 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.376723 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podStartSLOduration=4.376708638 podStartE2EDuration="4.376708638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:58.373202708 +0000 UTC m=+1356.355945119" watchObservedRunningTime="2026-10-31 23:12:58.376708638 +0000 UTC m=+1356.359451016" Oct 31 23:12:58.758071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.749462 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:12:58.770713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.770334 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:12:58.770713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:58.770405 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:12:59.758017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:59.750354 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:12:59.758017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:12:59.750413 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:13:00.757026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:00.754781 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:13:00.757026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:00.754849 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:13:01.762016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:01.755492 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:13:01.762016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:01.755569 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:13:02.754012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:02.749527 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:13:02.754309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:02.754278 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:13:03.752074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:03.748358 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:13:03.752074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:03.748437 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 23:13:05.164915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:05.164872 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:13:05.165469 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:05.165318 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:13:05.166315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:05.166282 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:13:09.479809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.479779 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:13:09.486574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.486547 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.507227 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.507192 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:13:09.628089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.628055 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:13:09.632954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.632926 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.648373 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648338 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648442 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648510 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648542 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648567 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648590 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648738 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648757 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.648858 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.648805 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvrh\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.654273 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.654247 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:13:09.750670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.750636 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.750839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.750703 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.750839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.750797 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.750954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.750840 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbfw\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.751033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.750957 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751034 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751069 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751094 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751217 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751351 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751360 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751392 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751421 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751424 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751462 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.751568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751551 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.752089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751607 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvrh\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.752089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751746 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.752089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.751983 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.752089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.752043 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.752362 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.752341 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.752431 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.752370 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.752525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.752508 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.753870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.753842 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.754241 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.754221 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.763569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.763543 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.764054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.764033 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvrh\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.812335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.812305 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:09.854435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854400 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854558 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854642 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854627 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854684 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854718 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbfw\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854823 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854807 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854895 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.854974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.854947 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855096 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855003 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855096 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855032 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855358 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855420 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855424 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.855738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.855650 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.857331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.857308 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.857513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.857486 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.868680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.868645 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbfw\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.869179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.869156 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:09.949549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:09.949515 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:10.124140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.123675 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:13:10.124505 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:13:10.124467 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4421fe12_f738_4e07_9463_a15933118db0.slice/crio-af00602279314a65473773985eea8b6e8d31ba2ea40865f4b7ab466fbd87e38a WatchSource:0}: Error finding container af00602279314a65473773985eea8b6e8d31ba2ea40865f4b7ab466fbd87e38a: Status 404 returned error can't find the container with id af00602279314a65473773985eea8b6e8d31ba2ea40865f4b7ab466fbd87e38a Oct 31 23:13:10.127212 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.127179 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.127308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.127246 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.127308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.127282 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.290659 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.290572 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:13:10.300356 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:13:10.300316 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ff1268_d948_4eb5_807b_3e4f9c548f1b.slice/crio-4872701db1966ce47475700cb85582a78377f4e78a33fc844a370eea39d2e2ae WatchSource:0}: Error finding container 4872701db1966ce47475700cb85582a78377f4e78a33fc844a370eea39d2e2ae: Status 404 returned error can't find the container with id 4872701db1966ce47475700cb85582a78377f4e78a33fc844a370eea39d2e2ae Oct 31 23:13:10.303943 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.303903 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.304087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.303984 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.304172 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.304102 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:10.512359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.512318 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" event={"ID":"4421fe12-f738-4e07-9463-a15933118db0","Type":"ContainerStarted","Data":"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c"} Oct 31 23:13:10.512856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.512383 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" event={"ID":"4421fe12-f738-4e07-9463-a15933118db0","Type":"ContainerStarted","Data":"af00602279314a65473773985eea8b6e8d31ba2ea40865f4b7ab466fbd87e38a"} Oct 31 23:13:10.519091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.519060 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" event={"ID":"82ff1268-d948-4eb5-807b-3e4f9c548f1b","Type":"ContainerStarted","Data":"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547"} Oct 31 23:13:10.519250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.519117 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" event={"ID":"82ff1268-d948-4eb5-807b-3e4f9c548f1b","Type":"ContainerStarted","Data":"4872701db1966ce47475700cb85582a78377f4e78a33fc844a370eea39d2e2ae"} Oct 31 23:13:10.542189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.542096 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podStartSLOduration=1.5420824419999999 podStartE2EDuration="1.542082442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.535512353 +0000 UTC m=+1368.518254733" watchObservedRunningTime="2026-10-31 23:13:10.542082442 +0000 UTC m=+1368.524824820" Oct 31 23:13:10.564073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.564006 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podStartSLOduration=1.5639706420000001 podStartE2EDuration="1.563970642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.559726659 +0000 UTC m=+1368.542469039" watchObservedRunningTime="2026-10-31 23:13:10.563970642 +0000 UTC m=+1368.546713022" Oct 31 23:13:10.813072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.812975 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:10.960014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:10.952090 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:11.814110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:11.814074 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:11.814499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:11.814128 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:11.966026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:11.952867 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:11.966026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:11.952929 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:12.812833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:12.812798 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.813197 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:12.812853 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:12.983011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:12.977105 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.983011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:12.977168 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:13.813658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:13.813552 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:13.813658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:13.813601 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:13.957023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:13.952429 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:13.957023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:13.952495 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:14.813236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:14.813196 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:14.813639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:14.813261 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:14.918948 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:13:14.918919 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:13:14.962016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:14.952420 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:14.962016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:14.952480 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:15.812962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:15.812927 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.813134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:15.812980 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:15.956013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:15.951427 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.956013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:15.951491 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:16.855706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:16.855258 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:16.855706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:16.855320 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:16.955184 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:16.955133 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:16.955503 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:16.955458 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:17.861521 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:17.861483 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:13:17.897808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:17.897766 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 23:13:17.960105 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:17.960052 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:17.960455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:17.960423 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:19.092041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.073149 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 23:13:19.092041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.073213 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 23:13:19.105555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.105514 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:13:19.152108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.152062 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.175216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.175169 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.263800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.263258 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.322041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.314120 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:13:19.375171 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.375071 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.431597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.431545 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.652939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.652569 7074 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:19.652939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.652628 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.908407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.907632 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:19.908407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:19.907700 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.132782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.124095 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.143176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.134323 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:13:20.183181 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.183059 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.200562 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.194648 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 31 23:13:20.200562 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.195913 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.245478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.245380 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.272656 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.264075 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:13:20.287214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.287095 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.322944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.322105 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.387041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.383022 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.470065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.469195 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.913018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.912776 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:20.913018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:20.912852 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.225464 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:13:21.225372 7074 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.066s" Oct 31 23:13:21.226212 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.226192 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:13:21.226396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.226382 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:13:21.226580 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.226565 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:13:21.226765 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.226750 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:13:21.226930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.226917 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:13:21.229124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.229103 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:13:21.229314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.229299 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:13:21.284268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.284223 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:13:21.356693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.314362 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.356926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.328225 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.357075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.344632 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:21.357214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.357188 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.608042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.607970 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:13:21.656011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.646889 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.761806 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.760216 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:13:21.945200 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:21.937131 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:13:22.006436 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.006408 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:13:22.010897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.010875 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:13:22.039101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.037772 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:22.039101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.037828 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.039101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.038434 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:13:22.043054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.043029 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:13:22.050847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.050825 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:13:22.358655 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.358623 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:22.359189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.359159 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.594067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.594031 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:22.595342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.595311 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:22.597962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:22.597935 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:13:23.312565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:23.312531 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:23.314186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:23.314157 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:23.314839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:23.314818 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:13:39.673907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.673860 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:13:39.683315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.683239 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.763667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.763638 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:13:39.842054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842014 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842082 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzm7v\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842217 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842333 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842526 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842594 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842638 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842726 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.842798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.842767 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.943977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.943883 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.943977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.943950 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.943977 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.943980 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944079 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944112 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944163 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944199 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzm7v\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944227 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944307 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944404 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944477 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944632 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944820 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.944984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.944957 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.947001 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.946964 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.947001 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.946983 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.956409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.956385 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:39.956541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:39.956523 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzm7v\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:40.067320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:40.067284 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:40.360971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:40.360944 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:13:40.361267 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:13:40.361246 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6650d91_acb8_4428_93e2_fbeea25cbd13.slice/crio-9c9bbfb584192efd70592f9371873e56d0334268eb3726b63d1838f89c501a35 WatchSource:0}: Error finding container 9c9bbfb584192efd70592f9371873e56d0334268eb3726b63d1838f89c501a35: Status 404 returned error can't find the container with id 9c9bbfb584192efd70592f9371873e56d0334268eb3726b63d1838f89c501a35 Oct 31 23:13:40.363868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:40.363833 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:40.363960 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:40.363915 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:40.363960 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:40.363943 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:13:41.122779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:41.122739 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" event={"ID":"d6650d91-acb8-4428-93e2-fbeea25cbd13","Type":"ContainerStarted","Data":"979c62b871b897cb3eadd0b1d8e1593c994a1acfad68490ddda48bd02fe59eb1"} Oct 31 23:13:41.123324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:41.122799 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" event={"ID":"d6650d91-acb8-4428-93e2-fbeea25cbd13","Type":"ContainerStarted","Data":"9c9bbfb584192efd70592f9371873e56d0334268eb3726b63d1838f89c501a35"} Oct 31 23:13:42.073015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:42.072332 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:42.078145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:42.077189 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:42.078145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:42.077272 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:43.077010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:43.071398 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:43.077010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:43.071473 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:44.068812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:44.068773 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:44.069177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:44.068840 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:44.897893 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:13:44.897853 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:13:45.073461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:45.073423 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:45.073649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:45.073512 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:46.073355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:46.073317 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:46.073890 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:46.073388 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:47.072213 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:47.072179 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:47.072506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:47.072479 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:48.075027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:48.070822 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:48.083189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:48.083148 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:48.945556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:48.945111 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.032392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.011312 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.032392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.018067 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.055043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.052914 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.064034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.060030 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.129014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.068646 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 23:13:49.129014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.068704 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 23:13:49.152188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.152137 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.014046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:49.980252 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.018403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.018340 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.091352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.051531 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:13:50.146026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.060651 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.378077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.147502 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:13:50.571894 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.571860 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:13:50.704073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.220011 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:13:50.704310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.465231 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 31 23:13:50.704427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.465436 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.709011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.465570 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.709196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.219100 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.777754 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.777717 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:13:50.778608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.778583 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:13:50.863276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.863219 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qw4q4" podUID="b829b94d-8617-461c-9338-114f98be38b9" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.38:8443/ready\": context deadline exceeded" Oct 31 23:13:50.985074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.984080 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.994014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.985362 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:50.994014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:50.985409 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.039059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.029717 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podStartSLOduration=12.029685286 podStartE2EDuration="12.029685286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:41.151882812 +0000 UTC m=+1399.134625193" watchObservedRunningTime="2026-10-31 23:13:51.029685286 +0000 UTC m=+1409.012427665" Oct 31 23:13:51.180177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.180132 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.245265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.236355 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:13:51.332308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.332258 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.369415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.368216 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.414028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.398146 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:51.414028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.398211 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.451384 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.451317 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:51.451659 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.451598 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.792037 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:13:51.790557 7074 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.012s" Oct 31 23:13:51.792037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.790652 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:13:51.792037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.790710 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:13:51.792037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.790797 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:13:51.900144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.885461 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.926387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:51.926324 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.016409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.003065 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.222108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.191181 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:13:52.237196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.237142 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.365157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.365104 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.437015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.436157 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:13:52.480208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.480106 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:52.484106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.484062 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.897017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.896148 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:13:52.931212 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:52.931164 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.031013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.030250 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.060015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.056132 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:53.086441 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.086409 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:53.136059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.125562 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:13:53.191448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.191322 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:13:53.253470 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.253414 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:13:53.434014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.433610 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.460276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.460171 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.486327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.476517 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:13:53.730071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.701964 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.766587 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.766558 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:13:53.792109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.792068 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:13:53.797646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.779796 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.897451 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.897424 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:13:53.951752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.951709 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:13:53.981905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:53.981818 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:13:54.008395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:13:54.008360 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:14:05.957713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:05.957637 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.018627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:06.018579 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.275024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:06.272037 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.364011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:05.970191 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.394156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:06.394105 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.395817 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:06.394402 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:14:06.737072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:06.736136 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.006069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.005284 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.006069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.005937 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.006652 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.006315 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:14:07.044047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.040162 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:14:07.044047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.040701 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:14:07.288073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.281565 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:14:07.288073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.282434 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:14:07.401152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.401099 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.578194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.551263 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:14:07.578194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.429147 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.578194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.480028 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.578194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.567812 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:07.578194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:07.567875 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:14:08.015565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.013365 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:14:08.015565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.015349 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:14:08.030381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.030328 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:08.187017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.179573 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:14:08.479850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.467063 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:08.595017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.594952 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:14:08.999899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:08.833235 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.063272 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.063226 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.064422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.048251 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:09.064703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.050264 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.172148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.168605 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:14:09.221176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.221122 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.226035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.224685 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.237806 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.237760 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:14:09.480076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.431138 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.480076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.431211 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.505751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.505695 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.673489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.673440 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:14:09.718265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.718196 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.819015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:09.794769 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:14:10.179563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.179181 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.286299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.286227 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:14:10.286961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.286913 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.343191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.343135 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.452959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.452856 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.462384 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.461208 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.530014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.506444 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.694536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.694488 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:14:10.729617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.729478 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.951009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:10.950053 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:14:11.195280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.195244 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:11.219015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.215135 7074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 23:14:11.499075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.329311 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:14:11.569288 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.329349 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:14:11.601033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.329392 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:14:11.601033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.539505 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.710160 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.710111 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.749458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.749347 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.952211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:11.952146 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.308585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.308547 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:14:12.458294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.457833 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:14:12.531117 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.531085 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:14:12.601736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.598271 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:14:12.601736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.598921 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.603915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.603216 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.606515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.605442 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:12.625942 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.625915 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:14:12.627196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:12.627166 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:14:14.894888 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:14:14.894844 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:14:32.472145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:32.472078 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:14:32.629521 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:32.629474 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:14:32.759152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:32.755143 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.207513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.198386 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.303334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.303289 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:14:33.605624 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.605570 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.639133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.625624 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:14:33.649900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.627347 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.650267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.634628 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.713055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.684462 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:33.722282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.722243 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 31 23:14:33.729019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.728282 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.756743 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.756510 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.757651 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:33.757619 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:14:34.021737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.021682 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.403139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.022018 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:34.403395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.403363 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.403529 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.205010 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:14:34.404746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.094120 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.405892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.095115 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:34.406074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.406049 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.410013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.406444 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:14:34.471152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.471102 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.497617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.473496 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:14:34.645380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.645327 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.694686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.694590 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.166012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.755358 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.193012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.180641 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:14:35.334014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:34.782716 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:14:35.449295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.449195 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.455529 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.455505 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:14:35.485044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.452801 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.485044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.479875 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.736070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:35.680621 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:14:36.264732 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.262133 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:14:36.264732 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.262539 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:14:36.332017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.330096 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.470021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.468105 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.499006 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.482184 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.499006 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.492239 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:14:36.504389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.504348 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.515250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.515054 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:14:36.518811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.518180 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:14:36.519672 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.519114 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:14:36.552356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.552322 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:14:36.568899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.568284 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:14:36.614029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:36.610878 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:14:39.376103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.376069 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:14:39.383359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.383332 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.390598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.390566 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b2c6553b-serving-cert\"" Oct 31 23:14:39.402077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.402055 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:14:39.483753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.483714 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.483961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.483933 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.484122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.483998 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvzg\" (UniqueName: \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.484122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.484065 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.585934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.585904 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.586131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.585958 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvzg\" (UniqueName: \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.586131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.586033 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.586242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.586128 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.586607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.586585 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.588604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.588579 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.588701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.588605 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.600922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.600900 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvzg\" (UniqueName: \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\") pod \"maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:39.708822 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:39.708728 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:40.201568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:40.201528 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:14:40.203631 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:14:40.203596 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756f4caa_1568_4430_b3a1_0ecfadaa4f4d.slice/crio-b93c366f4c84c511f413e4d2ae10912fcf449f5656d740b73c0ae215f80f29bb WatchSource:0}: Error finding container b93c366f4c84c511f413e4d2ae10912fcf449f5656d740b73c0ae215f80f29bb: Status 404 returned error can't find the container with id b93c366f4c84c511f413e4d2ae10912fcf449f5656d740b73c0ae215f80f29bb Oct 31 23:14:40.592179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:40.591543 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" event={"ID":"756f4caa-1568-4430-b3a1-0ecfadaa4f4d","Type":"ContainerStarted","Data":"b93c366f4c84c511f413e4d2ae10912fcf449f5656d740b73c0ae215f80f29bb"} Oct 31 23:14:41.600379 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:41.600346 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" event={"ID":"756f4caa-1568-4430-b3a1-0ecfadaa4f4d","Type":"ContainerStarted","Data":"601d0d377f57b66acbd15ab4ecfbde37184966ca7536b67eed09ea47ce1bf217"} Oct 31 23:14:41.600805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:41.600563 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:41.624963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:41.624903 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" podStartSLOduration=2.128272409 podStartE2EDuration="2.624889369s" totalImagesPullingTime="496.61696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:41.620632166 +0000 UTC m=+1459.603374554" watchObservedRunningTime="2026-10-31 23:14:41.624889369 +0000 UTC m=+1459.607631768" Oct 31 23:14:44.832222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.832182 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:14:44.855382 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.855346 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:14:44.855557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.855465 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:44.861750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.861705 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ac5ff20f-serving-cert\"" Oct 31 23:14:44.898510 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:14:44.898469 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:14:44.956099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.956066 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7pw\" (UniqueName: \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:44.956252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.956113 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:44.956342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.956328 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:44.956399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:44.956387 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.063014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.060952 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.063014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.061083 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.063014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.061256 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7pw\" (UniqueName: \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.063014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.061297 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.063371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.063133 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.066178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.066154 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.066660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.066637 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.075581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.075554 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7pw\" (UniqueName: \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\") pod \"maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.173937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.173854 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:45.541161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.541122 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:14:45.664383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:45.664343 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" event={"ID":"f41d7da3-7a17-4184-b200-0ab6ea97f8a8","Type":"ContainerStarted","Data":"9bb4ec3a7e04d7ab5970b9157ab03874ce854c90060c8550c0bd7ad640635186"} Oct 31 23:14:46.673719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:46.673677 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" event={"ID":"f41d7da3-7a17-4184-b200-0ab6ea97f8a8","Type":"ContainerStarted","Data":"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d"} Oct 31 23:14:46.674275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:46.673772 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:14:46.699216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:46.699164 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" podStartSLOduration=2.284484026 podStartE2EDuration="2.699148893s" totalImagesPullingTime="414.664867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:46.695437221 +0000 UTC m=+1464.678179603" watchObservedRunningTime="2026-10-31 23:14:46.699148893 +0000 UTC m=+1464.681891271" Oct 31 23:14:47.614424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:47.614395 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:14:52.692284 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:14:52.692258 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:15:00.355895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.355858 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:00.361178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.361155 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.393527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.393492 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:00.407051 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.407025 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:15:00.507688 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.507653 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.507888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.507776 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2mnl\" (UniqueName: \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.609881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.609790 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.609881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.609879 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2mnl\" (UniqueName: \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.610609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.610585 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.622821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.622793 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2mnl\" (UniqueName: \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\") pod \"maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.679822 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.679784 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:00.966700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:00.966672 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:00.967445 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:15:00.967420 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c85890_5ce6_44c2_8c50_a29bc0dfd154.slice/crio-8dd85ed63a36545a13c26f493968566e47b16bcefcc41ac3d1c348fe6da5d846 WatchSource:0}: Error finding container 8dd85ed63a36545a13c26f493968566e47b16bcefcc41ac3d1c348fe6da5d846: Status 404 returned error can't find the container with id 8dd85ed63a36545a13c26f493968566e47b16bcefcc41ac3d1c348fe6da5d846 Oct 31 23:15:01.870211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:01.870046 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerStarted","Data":"8dd85ed63a36545a13c26f493968566e47b16bcefcc41ac3d1c348fe6da5d846"} Oct 31 23:15:05.695417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.695364 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:15:05.696066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.696029 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" podUID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" containerName="maas-api" containerID="cri-o://601d0d377f57b66acbd15ab4ecfbde37184966ca7536b67eed09ea47ce1bf217" gracePeriod=30 Oct 31 23:15:05.927444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.927226 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerStarted","Data":"c07030e445b149e0f90216bf3485a7908b59a191eee885746a228cbc83ec30c4"} Oct 31 23:15:05.945342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.945313 7074 generic.go:356] "Generic (PLEG): container finished" podID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" containerID="601d0d377f57b66acbd15ab4ecfbde37184966ca7536b67eed09ea47ce1bf217" exitCode=0 Oct 31 23:15:05.945520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.945395 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" event={"ID":"756f4caa-1568-4430-b3a1-0ecfadaa4f4d","Type":"ContainerDied","Data":"601d0d377f57b66acbd15ab4ecfbde37184966ca7536b67eed09ea47ce1bf217"} Oct 31 23:15:05.965893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:05.965496 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" podStartSLOduration=2.060758696 podStartE2EDuration="5.965461192s" totalImagesPullingTime="3.904702496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:05.962671453 +0000 UTC m=+1483.945413832" watchObservedRunningTime="2026-10-31 23:15:05.965461192 +0000 UTC m=+1483.948203572" Oct 31 23:15:06.017597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.017573 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:15:06.084116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.083169 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\") pod \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " Oct 31 23:15:06.084116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.083280 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\") pod \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " Oct 31 23:15:06.084116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.083398 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\" (UniqueName: \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\") pod \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " Oct 31 23:15:06.084116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.083512 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\") pod \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\" (UID: \"756f4caa-1568-4430-b3a1-0ecfadaa4f4d\") " Oct 31 23:15:06.085049 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.084909 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca" pod "756f4caa-1568-4430-b3a1-0ecfadaa4f4d" (UID: "756f4caa-1568-4430-b3a1-0ecfadaa4f4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:06.093060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.090008 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls" pod "756f4caa-1568-4430-b3a1-0ecfadaa4f4d" (UID: "756f4caa-1568-4430-b3a1-0ecfadaa4f4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:06.093060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.090152 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg" pod "756f4caa-1568-4430-b3a1-0ecfadaa4f4d" (UID: "756f4caa-1568-4430-b3a1-0ecfadaa4f4d"). InnerVolumeSpecName "kube-api-access-lfvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:06.103109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.096471 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:06.103109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.097908 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs" pod "756f4caa-1568-4430-b3a1-0ecfadaa4f4d" (UID: "756f4caa-1568-4430-b3a1-0ecfadaa4f4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:06.262018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.259823 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvzg\" (UniqueName: \"kubernetes.io/projected/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-kube-api-access-lfvzg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:06.262018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.259859 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:06.262018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:06.259874 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756f4caa-1568-4430-b3a1-0ecfadaa4f4d-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:07.360115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.347615 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" event={"ID":"756f4caa-1568-4430-b3a1-0ecfadaa4f4d","Type":"ContainerDied","Data":"b93c366f4c84c511f413e4d2ae10912fcf449f5656d740b73c0ae215f80f29bb"} Oct 31 23:15:07.360115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.347711 7074 scope.go:122] "RemoveContainer" containerID="601d0d377f57b66acbd15ab4ecfbde37184966ca7536b67eed09ea47ce1bf217" Oct 31 23:15:07.378037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.370612 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh" Oct 31 23:15:07.640074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.597551 7074 generic.go:356] "Generic (PLEG): container finished" podID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerID="c07030e445b149e0f90216bf3485a7908b59a191eee885746a228cbc83ec30c4" exitCode=60 Oct 31 23:15:07.640074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.597617 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerDied","Data":"c07030e445b149e0f90216bf3485a7908b59a191eee885746a228cbc83ec30c4"} Oct 31 23:15:07.640074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.598146 7074 scope.go:122] "RemoveContainer" containerID="c07030e445b149e0f90216bf3485a7908b59a191eee885746a228cbc83ec30c4" Oct 31 23:15:07.741124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.741080 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:15:07.873131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:07.873090 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b2c6553b-74cd77f9f-q8qvh"] Oct 31 23:15:08.255073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.250963 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.279237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.279190 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.491285 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.491154 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.498741 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.498683 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.531177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.522138 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 23:15:08.531366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.531242 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" path="/var/lib/kubelet/pods/756f4caa-1568-4430-b3a1-0ecfadaa4f4d/volumes" Oct 31 23:15:08.540018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.531588 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.786504 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:08.786411 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 23:15:09.266191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.266151 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:09.266473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.266422 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.287015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.284069 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.287015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.284494 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:15:09.519903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.506142 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.519903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.506738 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:15:09.636015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.633895 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.636015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:09.634354 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:15:10.315149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.214149 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.315149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.240181 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.315149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.313095 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 23:15:10.335506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.334122 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 23:15:10.364244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.336256 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 31 23:15:10.364244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.338561 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 23:15:10.507930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.507885 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:15:10.592337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.529835 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.687035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.683221 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.821427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.821373 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.946098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:10.920137 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerStarted","Data":"feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce"} Oct 31 23:15:11.318457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.318403 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.494056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.372546 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.494056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.420975 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:15:11.494056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.426097 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.494056 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.427367 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:15:11.652073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.575285 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.699108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.592126 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:15:11.699108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.665800 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:15:11.699108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.688928 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.831015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.830634 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.980966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:11.852507 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:15:12.331287 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.328710 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.384458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.379590 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p76fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.384458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.379682 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.384458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.380027 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.408481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.408445 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dvbhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.408712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.408686 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.441555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.441517 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mcdvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.441830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.441800 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.472012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.442697 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.511027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.504017 7074 generic.go:356] "Generic (PLEG): container finished" podID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerID="feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce" exitCode=60 Oct 31 23:15:12.511027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.504667 7074 scope.go:122] "RemoveContainer" containerID="feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce" Oct 31 23:15:12.511027 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:12.505147 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl_odh-ai-gateway-infra(f1c85890-5ce6-44c2-8c50-a29bc0dfd154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" Oct 31 23:15:12.714876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.695541 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.714876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.696450 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 23:15:12.771027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.759517 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 23:15:12.873239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.873201 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerDied","Data":"feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce"} Oct 31 23:15:12.886132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.886103 7074 scope.go:122] "RemoveContainer" containerID="c07030e445b149e0f90216bf3485a7908b59a191eee885746a228cbc83ec30c4" Oct 31 23:15:12.887466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:12.887434 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.318611 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.318282 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:15:13.332139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.332101 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.438468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.438388 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.531075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.515546 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:15:13.622038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.621117 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.696164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.696116 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.737371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.737325 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:15:13.739011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.738971 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:15:13.751522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.751488 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.766104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.761199 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" podUID="937794d1-b7e8-4afe-8915-c4916cd7c9cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.766104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.761529 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.775492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.773787 7074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.787225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.787115 7074 scope.go:122] "RemoveContainer" containerID="feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce" Oct 31 23:15:13.787816 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:13.787769 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl_odh-ai-gateway-infra(f1c85890-5ce6-44c2-8c50-a29bc0dfd154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" Oct 31 23:15:13.943122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.943090 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr" Oct 31 23:15:13.971207 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.971180 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:15:13.971835 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:13.971813 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:15:14.893812 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:14.893772 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:15:15.767039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:15.766983 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:15:15.767344 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:15.767262 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" podUID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" containerName="maas-api" containerID="cri-o://c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d" gracePeriod=30 Oct 31 23:15:15.821670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:15.821130 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:16.258468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.256510 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:16.349805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.346035 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\" (UniqueName: \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\") pod \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " Oct 31 23:15:16.349805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.346204 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\") pod \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\" (UID: \"f1c85890-5ce6-44c2-8c50-a29bc0dfd154\") " Oct 31 23:15:16.350174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.350066 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca" pod "f1c85890-5ce6-44c2-8c50-a29bc0dfd154" (UID: "f1c85890-5ce6-44c2-8c50-a29bc0dfd154"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:16.354071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.350661 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.371028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.370952 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl" pod "f1c85890-5ce6-44c2-8c50-a29bc0dfd154" (UID: "f1c85890-5ce6-44c2-8c50-a29bc0dfd154"). InnerVolumeSpecName "kube-api-access-c2mnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:16.441595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.441564 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:15:16.458341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.452952 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2mnl\" (UniqueName: \"kubernetes.io/projected/f1c85890-5ce6-44c2-8c50-a29bc0dfd154-kube-api-access-c2mnl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.568787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.559764 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\") pod \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " Oct 31 23:15:16.568787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.559831 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\") pod \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " Oct 31 23:15:16.568787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.559905 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\") pod \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " Oct 31 23:15:16.568787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.559942 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\" (UniqueName: \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\") pod \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\" (UID: \"f41d7da3-7a17-4184-b200-0ab6ea97f8a8\") " Oct 31 23:15:16.572363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.571211 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca" pod "f41d7da3-7a17-4184-b200-0ab6ea97f8a8" (UID: "f41d7da3-7a17-4184-b200-0ab6ea97f8a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:16.594637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.594539 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw" pod "f41d7da3-7a17-4184-b200-0ab6ea97f8a8" (UID: "f41d7da3-7a17-4184-b200-0ab6ea97f8a8"). InnerVolumeSpecName "kube-api-access-7l7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:16.594637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.594542 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs" pod "f41d7da3-7a17-4184-b200-0ab6ea97f8a8" (UID: "f41d7da3-7a17-4184-b200-0ab6ea97f8a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:16.599152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.599095 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls" pod "f41d7da3-7a17-4184-b200-0ab6ea97f8a8" (UID: "f41d7da3-7a17-4184-b200-0ab6ea97f8a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:16.662428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.662330 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.662428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.662369 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7pw\" (UniqueName: \"kubernetes.io/projected/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-kube-api-access-7l7pw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.662428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.662387 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.662428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.662403 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41d7da3-7a17-4184-b200-0ab6ea97f8a8-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.826322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.826286 7074 generic.go:356] "Generic (PLEG): container finished" podID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" containerID="c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d" exitCode=0 Oct 31 23:15:16.826515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.826350 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" Oct 31 23:15:16.826515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.826374 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" event={"ID":"f41d7da3-7a17-4184-b200-0ab6ea97f8a8","Type":"ContainerDied","Data":"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d"} Oct 31 23:15:16.826515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.826448 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8" event={"ID":"f41d7da3-7a17-4184-b200-0ab6ea97f8a8","Type":"ContainerDied","Data":"9bb4ec3a7e04d7ab5970b9157ab03874ce854c90060c8550c0bd7ad640635186"} Oct 31 23:15:16.826515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.826482 7074 scope.go:122] "RemoveContainer" containerID="c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d" Oct 31 23:15:16.839499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.833356 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" event={"ID":"f1c85890-5ce6-44c2-8c50-a29bc0dfd154","Type":"ContainerDied","Data":"8dd85ed63a36545a13c26f493968566e47b16bcefcc41ac3d1c348fe6da5d846"} Oct 31 23:15:16.839499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.833404 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl" Oct 31 23:15:16.844609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.844559 7074 scope.go:122] "RemoveContainer" containerID="c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d" Oct 31 23:15:16.844936 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:16.844907 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d\": container with ID starting with c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d not found: ID does not exist" containerID="c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d" Oct 31 23:15:16.845046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.844933 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d"} err="failed to get container status \"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d\": rpc error: code = NotFound desc = could not find container \"c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d\": container with ID starting with c221ab2986fda59ac5e6c04527e4c1a47b047dd163231a18b47182e059b7e41d not found: ID does not exist" Oct 31 23:15:16.845046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.844956 7074 scope.go:122] "RemoveContainer" containerID="feb64258aee37605375ac01d7a89101af5b55474f43d47cee0ae43b49b18c3ce" Oct 31 23:15:16.863653 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.863623 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:15:16.868192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.868165 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac5ff20f-5655ff5c68-r8cq8"] Oct 31 23:15:16.880795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.880762 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:16.887192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:16.887170 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ac5ff20f-29891475-xngxl"] Oct 31 23:15:18.146520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:18.146488 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" path="/var/lib/kubelet/pods/f1c85890-5ce6-44c2-8c50-a29bc0dfd154/volumes" Oct 31 23:15:18.147063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:18.147047 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" path="/var/lib/kubelet/pods/f41d7da3-7a17-4184-b200-0ab6ea97f8a8/volumes" Oct 31 23:15:21.040183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.040143 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:21.041231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041204 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.041231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041229 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041252 7074 cpu_manager.go:414] "containerMap: removing container" podUID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" containerName="maas-api" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041270 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" containerName="maas-api" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041289 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" containerName="maas-api" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041296 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" containerName="maas-api" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041349 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.041417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041357 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.041702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041533 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.041702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041557 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="756f4caa-1568-4430-b3a1-0ecfadaa4f4d" containerName="maas-api" Oct 31 23:15:21.041702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.041569 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41d7da3-7a17-4184-b200-0ab6ea97f8a8" containerName="maas-api" Oct 31 23:15:21.052537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.052509 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.056916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.056885 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:21.060973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.060824 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-56b15fa9\"" Oct 31 23:15:21.062139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.061553 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-56b15fa9-dockercfg-vblwq\"" Oct 31 23:15:21.092129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.092093 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:21.093092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.093069 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c85890-5ce6-44c2-8c50-a29bc0dfd154" containerName="cleanup" Oct 31 23:15:21.097635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.097613 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.110666 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.110625 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:21.259015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.255941 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.259015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.256313 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.259015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.256378 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8d5\" (UniqueName: \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.259015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.256410 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6xz\" (UniqueName: \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.367446 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8d5\" (UniqueName: \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.367507 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6xz\" (UniqueName: \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.367728 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.368052 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.369177 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.397078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.370757 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.416138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.413820 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6xz\" (UniqueName: \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\") pod \"payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.459953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.459916 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8d5\" (UniqueName: \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\") pod \"payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.483522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.483487 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:21.675042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.672969 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:21.830846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.830811 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:21.832342 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:15:21.832316 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0566a6d5_d6b4_492c_94ec_c55e33181b6c.slice/crio-77b18ea7c5fb51725235d8624e71b4a7892162b7baf7fac7be7c5810aa38d6b1 WatchSource:0}: Error finding container 77b18ea7c5fb51725235d8624e71b4a7892162b7baf7fac7be7c5810aa38d6b1: Status 404 returned error can't find the container with id 77b18ea7c5fb51725235d8624e71b4a7892162b7baf7fac7be7c5810aa38d6b1 Oct 31 23:15:21.887893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:21.887856 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" event={"ID":"0566a6d5-d6b4-492c-94ec-c55e33181b6c","Type":"ContainerStarted","Data":"77b18ea7c5fb51725235d8624e71b4a7892162b7baf7fac7be7c5810aa38d6b1"} Oct 31 23:15:22.004132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.004103 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:22.006260 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:15:22.006233 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c407c73_fca9_4ea5_82c4_d873e928678a.slice/crio-b8bbb102d2fa69548876e1834850fcf0506fd4c9d0d817ab41591ded26b414c1 WatchSource:0}: Error finding container b8bbb102d2fa69548876e1834850fcf0506fd4c9d0d817ab41591ded26b414c1: Status 404 returned error can't find the container with id b8bbb102d2fa69548876e1834850fcf0506fd4c9d0d817ab41591ded26b414c1 Oct 31 23:15:22.371354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.371275 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:15:22.372495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.372462 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:15:22.900016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.897509 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" event={"ID":"0566a6d5-d6b4-492c-94ec-c55e33181b6c","Type":"ContainerStarted","Data":"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354"} Oct 31 23:15:22.900016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.897599 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:22.900016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.899428 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" event={"ID":"3c407c73-fca9-4ea5-82c4-d873e928678a","Type":"ContainerStarted","Data":"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071"} Oct 31 23:15:22.900016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.899478 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" event={"ID":"3c407c73-fca9-4ea5-82c4-d873e928678a","Type":"ContainerStarted","Data":"b8bbb102d2fa69548876e1834850fcf0506fd4c9d0d817ab41591ded26b414c1"} Oct 31 23:15:22.900016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.899529 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:22.916794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.916720 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" podStartSLOduration=1.916705539 podStartE2EDuration="1.916705539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:22.915875494 +0000 UTC m=+1500.898617872" watchObservedRunningTime="2026-10-31 23:15:22.916705539 +0000 UTC m=+1500.899447917" Oct 31 23:15:22.944699 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:22.944651 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" podStartSLOduration=1.944635595 podStartE2EDuration="1.944635595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:22.939104973 +0000 UTC m=+1500.921847376" watchObservedRunningTime="2026-10-31 23:15:22.944635595 +0000 UTC m=+1500.927377972" Oct 31 23:15:27.421156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.421120 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:27.431034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.430973 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.435324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.435299 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-56b15fa9-serving-cert\"" Oct 31 23:15:27.436084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.436055 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:27.558861 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.558827 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.559149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.559126 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77pn\" (UniqueName: \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.559291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.559273 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.559398 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.559384 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.661625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.661593 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77pn\" (UniqueName: \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.661792 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.661715 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.661871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.661815 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.661938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.661912 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.662574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.662548 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.664426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.664403 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.664588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.664566 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.675225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.675172 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77pn\" (UniqueName: \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\") pod \"maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:27.750446 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:27.750408 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:28.072752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:28.072666 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:28.093309 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:15:28.093274 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84403b3b_4585_4f20_b981_473e90246330.slice/crio-6be62232d143744f7dd31d4eae551e197ee10e465ca814e34628d3022c696c50 WatchSource:0}: Error finding container 6be62232d143744f7dd31d4eae551e197ee10e465ca814e34628d3022c696c50: Status 404 returned error can't find the container with id 6be62232d143744f7dd31d4eae551e197ee10e465ca814e34628d3022c696c50 Oct 31 23:15:29.138059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:29.124345 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" event={"ID":"84403b3b-4585-4f20-b981-473e90246330","Type":"ContainerStarted","Data":"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f"} Oct 31 23:15:29.138059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:29.124414 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" event={"ID":"84403b3b-4585-4f20-b981-473e90246330","Type":"ContainerStarted","Data":"6be62232d143744f7dd31d4eae551e197ee10e465ca814e34628d3022c696c50"} Oct 31 23:15:29.138059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:29.127117 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:29.151615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:29.151557 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" podStartSLOduration=1.6783648690000001 podStartE2EDuration="2.151538754s" totalImagesPullingTime="473.173885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:29.150914401 +0000 UTC m=+1507.133656783" watchObservedRunningTime="2026-10-31 23:15:29.151538754 +0000 UTC m=+1507.134281136" Oct 31 23:15:30.316341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.316300 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:15:30.322880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.322293 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.328626 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.326933 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-907d182f-serving-cert\"" Oct 31 23:15:30.330124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.329917 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:15:30.454014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.432427 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp9w\" (UniqueName: \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.454014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.432597 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.454014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.432853 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.454014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.432906 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.534372 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.534335 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp9w\" (UniqueName: \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.534538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.534469 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.534730 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.534712 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.534793 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.534768 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.535512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.535487 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.537630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.537604 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.537818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.537662 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.548133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.548102 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp9w\" (UniqueName: \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\") pod \"maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.644145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.644042 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:30.978291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:30.972554 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:15:31.140955 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:31.140921 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" event={"ID":"0609dd55-fe7f-49d9-8b61-0dc25e694b5d","Type":"ContainerStarted","Data":"7380b8c9f95d7d311f2039b6f1149d6b3e7120afbb84e64ab69ff1030cf1e590"} Oct 31 23:15:32.149609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:32.149574 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" event={"ID":"0609dd55-fe7f-49d9-8b61-0dc25e694b5d","Type":"ContainerStarted","Data":"a0580f93666ccefdaa29134a8b7f681e5550f34f9c88d05e824ed0f4acb4107a"} Oct 31 23:15:32.150104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:32.149755 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:32.170877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:32.170830 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" podStartSLOduration=1.71562926 podStartE2EDuration="2.170815359s" totalImagesPullingTime="455.186099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:32.168960789 +0000 UTC m=+1510.151703191" watchObservedRunningTime="2026-10-31 23:15:32.170815359 +0000 UTC m=+1510.153557737" Oct 31 23:15:33.913353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:33.913317 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:33.913800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:33.913403 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:35.138265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:35.138237 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:38.162606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:38.162575 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:15:44.897419 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:44.897374 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:15:49.934506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:49.934470 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:49.940427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:49.940393 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:50.346626 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.346558 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:50.347122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.347071 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" podUID="84403b3b-4585-4f20-b981-473e90246330" containerName="maas-api" containerID="cri-o://f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f" gracePeriod=30 Oct 31 23:15:50.813958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.813931 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:50.947847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.947808 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\" (UniqueName: \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\") pod \"84403b3b-4585-4f20-b981-473e90246330\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " Oct 31 23:15:50.948347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.947919 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\") pod \"84403b3b-4585-4f20-b981-473e90246330\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " Oct 31 23:15:50.948347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.948102 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\") pod \"84403b3b-4585-4f20-b981-473e90246330\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " Oct 31 23:15:50.948347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.948265 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\") pod \"84403b3b-4585-4f20-b981-473e90246330\" (UID: \"84403b3b-4585-4f20-b981-473e90246330\") " Oct 31 23:15:50.952043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.951747 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca" pod "84403b3b-4585-4f20-b981-473e90246330" (UID: "84403b3b-4585-4f20-b981-473e90246330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:50.953664 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.953617 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn" pod "84403b3b-4585-4f20-b981-473e90246330" (UID: "84403b3b-4585-4f20-b981-473e90246330"). InnerVolumeSpecName "kube-api-access-n77pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:50.968408 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.968340 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs" pod "84403b3b-4585-4f20-b981-473e90246330" (UID: "84403b3b-4585-4f20-b981-473e90246330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:50.968760 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:50.968644 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls" pod "84403b3b-4585-4f20-b981-473e90246330" (UID: "84403b3b-4585-4f20-b981-473e90246330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:51.056132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.051310 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:51.056132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.051351 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84403b3b-4585-4f20-b981-473e90246330-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:51.056132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.051367 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84403b3b-4585-4f20-b981-473e90246330-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:51.056132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.051384 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77pn\" (UniqueName: \"kubernetes.io/projected/84403b3b-4585-4f20-b981-473e90246330-kube-api-access-n77pn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:51.465154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.465049 7074 generic.go:356] "Generic (PLEG): container finished" podID="84403b3b-4585-4f20-b981-473e90246330" containerID="f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f" exitCode=0 Oct 31 23:15:51.465154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.465099 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" Oct 31 23:15:51.465154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.465138 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" event={"ID":"84403b3b-4585-4f20-b981-473e90246330","Type":"ContainerDied","Data":"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f"} Oct 31 23:15:51.465426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.465195 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p" event={"ID":"84403b3b-4585-4f20-b981-473e90246330","Type":"ContainerDied","Data":"6be62232d143744f7dd31d4eae551e197ee10e465ca814e34628d3022c696c50"} Oct 31 23:15:51.465426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.465224 7074 scope.go:122] "RemoveContainer" containerID="f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f" Oct 31 23:15:51.481283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.481224 7074 scope.go:122] "RemoveContainer" containerID="f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f" Oct 31 23:15:51.481720 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:51.481689 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f\": container with ID starting with f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f not found: ID does not exist" containerID="f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f" Oct 31 23:15:51.481847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.481731 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f"} err="failed to get container status \"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f\": rpc error: code = NotFound desc = could not find container \"f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f\": container with ID starting with f242ac86471ac42d313a50c2c3e0358e25fb12215e18ec37a3bd3807b53f407f not found: ID does not exist" Oct 31 23:15:51.500816 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.500782 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:51.506191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:51.506159 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56b15fa9-7d8555c58c-9fw2p"] Oct 31 23:15:52.146701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:52.146662 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84403b3b-4585-4f20-b981-473e90246330" path="/var/lib/kubelet/pods/84403b3b-4585-4f20-b981-473e90246330/volumes" Oct 31 23:15:54.936360 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:54.936315 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" podUID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" containerName="payload-processing" containerID="cri-o://27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354" gracePeriod=25 Oct 31 23:15:54.940909 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:54.940703 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" podUID="3c407c73-fca9-4ea5-82c4-d873e928678a" containerName="payload-pre-processing" containerID="cri-o://685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071" gracePeriod=25 Oct 31 23:15:55.273614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.273288 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:55.356283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.356247 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:55.364935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.364903 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\") pod \"3c407c73-fca9-4ea5-82c4-d873e928678a\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " Oct 31 23:15:55.365144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.365027 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\" (UniqueName: \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\") pod \"3c407c73-fca9-4ea5-82c4-d873e928678a\" (UID: \"3c407c73-fca9-4ea5-82c4-d873e928678a\") " Oct 31 23:15:55.365422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.365390 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume" pod "3c407c73-fca9-4ea5-82c4-d873e928678a" (UID: "3c407c73-fca9-4ea5-82c4-d873e928678a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:55.366121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.366090 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c407c73-fca9-4ea5-82c4-d873e928678a-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.367616 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.367582 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz" pod "3c407c73-fca9-4ea5-82c4-d873e928678a" (UID: "3c407c73-fca9-4ea5-82c4-d873e928678a"). InnerVolumeSpecName "kube-api-access-dk6xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:55.467855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.467765 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\" (UniqueName: \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\") pod \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " Oct 31 23:15:55.467855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.467823 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\") pod \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\" (UID: \"0566a6d5-d6b4-492c-94ec-c55e33181b6c\") " Oct 31 23:15:55.468239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.468208 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume" pod "0566a6d5-d6b4-492c-94ec-c55e33181b6c" (UID: "0566a6d5-d6b4-492c-94ec-c55e33181b6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:55.468551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.468535 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0566a6d5-d6b4-492c-94ec-c55e33181b6c-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.468610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.468553 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6xz\" (UniqueName: \"kubernetes.io/projected/3c407c73-fca9-4ea5-82c4-d873e928678a-kube-api-access-dk6xz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.470310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.470283 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5" pod "0566a6d5-d6b4-492c-94ec-c55e33181b6c" (UID: "0566a6d5-d6b4-492c-94ec-c55e33181b6c"). InnerVolumeSpecName "kube-api-access-6z8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:55.509035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.508984 7074 generic.go:356] "Generic (PLEG): container finished" podID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" containerID="27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354" exitCode=0 Oct 31 23:15:55.509217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.509046 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" Oct 31 23:15:55.509217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.509121 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" event={"ID":"0566a6d5-d6b4-492c-94ec-c55e33181b6c","Type":"ContainerDied","Data":"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354"} Oct 31 23:15:55.509217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.509181 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh" event={"ID":"0566a6d5-d6b4-492c-94ec-c55e33181b6c","Type":"ContainerDied","Data":"77b18ea7c5fb51725235d8624e71b4a7892162b7baf7fac7be7c5810aa38d6b1"} Oct 31 23:15:55.509217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.509208 7074 scope.go:122] "RemoveContainer" containerID="27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354" Oct 31 23:15:55.511818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.511790 7074 generic.go:356] "Generic (PLEG): container finished" podID="3c407c73-fca9-4ea5-82c4-d873e928678a" containerID="685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071" exitCode=0 Oct 31 23:15:55.511913 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.511873 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" event={"ID":"3c407c73-fca9-4ea5-82c4-d873e928678a","Type":"ContainerDied","Data":"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071"} Oct 31 23:15:55.511981 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.511919 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" event={"ID":"3c407c73-fca9-4ea5-82c4-d873e928678a","Type":"ContainerDied","Data":"b8bbb102d2fa69548876e1834850fcf0506fd4c9d0d817ab41591ded26b414c1"} Oct 31 23:15:55.512058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.512006 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q" Oct 31 23:15:55.524875 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.524856 7074 scope.go:122] "RemoveContainer" containerID="27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354" Oct 31 23:15:55.525290 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:55.525261 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354\": container with ID starting with 27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354 not found: ID does not exist" containerID="27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354" Oct 31 23:15:55.525366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.525301 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354"} err="failed to get container status \"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354\": rpc error: code = NotFound desc = could not find container \"27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354\": container with ID starting with 27b00231d8da807017deced32284beb64498fa7869d83703f77df1ab1b078354 not found: ID does not exist" Oct 31 23:15:55.525366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.525327 7074 scope.go:122] "RemoveContainer" containerID="685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071" Oct 31 23:15:55.542858 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.542836 7074 scope.go:122] "RemoveContainer" containerID="685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071" Oct 31 23:15:55.543222 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:15:55.543199 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071\": container with ID starting with 685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071 not found: ID does not exist" containerID="685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071" Oct 31 23:15:55.543327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.543228 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071"} err="failed to get container status \"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071\": rpc error: code = NotFound desc = could not find container \"685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071\": container with ID starting with 685c8a604d274bcfc1f8b30fdf594a4e07bb3ee6c42f49b13939675fbaa79071 not found: ID does not exist" Oct 31 23:15:55.544315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.544298 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:55.549369 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.549347 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-56b15fa9-67c687ff74-5mrkh"] Oct 31 23:15:55.558164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.558139 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:55.564592 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.564569 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-56b15fa9-d84699fdc-9785q"] Oct 31 23:15:55.570706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:55.570684 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8d5\" (UniqueName: \"kubernetes.io/projected/0566a6d5-d6b4-492c-94ec-c55e33181b6c-kube-api-access-6z8d5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:56.144804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:56.144768 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" path="/var/lib/kubelet/pods/0566a6d5-d6b4-492c-94ec-c55e33181b6c/volumes" Oct 31 23:15:56.145274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:56.145258 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c407c73-fca9-4ea5-82c4-d873e928678a" path="/var/lib/kubelet/pods/3c407c73-fca9-4ea5-82c4-d873e928678a/volumes" Oct 31 23:15:57.717928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.717895 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:15:57.718824 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718806 7074 cpu_manager.go:414] "containerMap: removing container" podUID="3c407c73-fca9-4ea5-82c4-d873e928678a" containerName="payload-pre-processing" Oct 31 23:15:57.718892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718826 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c407c73-fca9-4ea5-82c4-d873e928678a" containerName="payload-pre-processing" Oct 31 23:15:57.718892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718843 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" containerName="payload-processing" Oct 31 23:15:57.718892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718848 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" containerName="payload-processing" Oct 31 23:15:57.718892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718862 7074 cpu_manager.go:414] "containerMap: removing container" podUID="84403b3b-4585-4f20-b981-473e90246330" containerName="maas-api" Oct 31 23:15:57.718892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718867 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84403b3b-4585-4f20-b981-473e90246330" containerName="maas-api" Oct 31 23:15:57.719100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718969 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c407c73-fca9-4ea5-82c4-d873e928678a" containerName="payload-pre-processing" Oct 31 23:15:57.719100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.718979 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0566a6d5-d6b4-492c-94ec-c55e33181b6c" containerName="payload-processing" Oct 31 23:15:57.719100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.719003 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="84403b3b-4585-4f20-b981-473e90246330" containerName="maas-api" Oct 31 23:15:57.724618 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.724597 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:57.735321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.735297 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:15:57.811333 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.811292 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8e6a08\"" Oct 31 23:15:57.811554 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.811457 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8e6a08-dockercfg-dn8jq\"" Oct 31 23:15:57.863015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.838353 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9s8\" (UniqueName: \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:57.863500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:57.863467 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.016014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.006398 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.016014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.006729 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9s8\" (UniqueName: \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.016014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.007973 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.064657 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.064568 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9s8\" (UniqueName: \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\") pod \"payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.345101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.344154 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:58.652704 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:58.652677 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:15:58.653152 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:15:58.653127 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01be9bb2_abc4_402c_84c9_8ea1f9c6d267.slice/crio-257cfe135e7afdbe6a58202edef69c87b80cf9208aa7806d0afb077f0e9d2ff1 WatchSource:0}: Error finding container 257cfe135e7afdbe6a58202edef69c87b80cf9208aa7806d0afb077f0e9d2ff1: Status 404 returned error can't find the container with id 257cfe135e7afdbe6a58202edef69c87b80cf9208aa7806d0afb077f0e9d2ff1 Oct 31 23:15:59.572762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:59.572362 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" event={"ID":"01be9bb2-abc4-402c-84c9-8ea1f9c6d267","Type":"ContainerStarted","Data":"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6"} Oct 31 23:15:59.572762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:59.572435 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" event={"ID":"01be9bb2-abc4-402c-84c9-8ea1f9c6d267","Type":"ContainerStarted","Data":"257cfe135e7afdbe6a58202edef69c87b80cf9208aa7806d0afb077f0e9d2ff1"} Oct 31 23:15:59.594012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:59.573592 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:15:59.604084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:15:59.601692 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" podStartSLOduration=2.601669357 podStartE2EDuration="2.601669357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:59.594797812 +0000 UTC m=+1537.577540191" watchObservedRunningTime="2026-10-31 23:15:59.601669357 +0000 UTC m=+1537.584411736" Oct 31 23:16:01.052529 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.049856 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:16:01.052529 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.050168 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" podUID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" containerName="maas-api" containerID="cri-o://a0580f93666ccefdaa29134a8b7f681e5550f34f9c88d05e824ed0f4acb4107a" gracePeriod=30 Oct 31 23:16:01.592702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.592659 7074 generic.go:356] "Generic (PLEG): container finished" podID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" containerID="a0580f93666ccefdaa29134a8b7f681e5550f34f9c88d05e824ed0f4acb4107a" exitCode=0 Oct 31 23:16:01.592866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.592705 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" event={"ID":"0609dd55-fe7f-49d9-8b61-0dc25e694b5d","Type":"ContainerDied","Data":"a0580f93666ccefdaa29134a8b7f681e5550f34f9c88d05e824ed0f4acb4107a"} Oct 31 23:16:01.592866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.592835 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" event={"ID":"0609dd55-fe7f-49d9-8b61-0dc25e694b5d","Type":"ContainerDied","Data":"7380b8c9f95d7d311f2039b6f1149d6b3e7120afbb84e64ab69ff1030cf1e590"} Oct 31 23:16:01.592866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.592857 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7380b8c9f95d7d311f2039b6f1149d6b3e7120afbb84e64ab69ff1030cf1e590" Oct 31 23:16:01.595852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.595830 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:16:01.658203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.658166 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\") pod \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " Oct 31 23:16:01.658432 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.658410 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\") pod \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " Oct 31 23:16:01.658533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.658488 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\") pod \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " Oct 31 23:16:01.658716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.658689 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\" (UniqueName: \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\") pod \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\" (UID: \"0609dd55-fe7f-49d9-8b61-0dc25e694b5d\") " Oct 31 23:16:01.659258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.659231 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca" pod "0609dd55-fe7f-49d9-8b61-0dc25e694b5d" (UID: "0609dd55-fe7f-49d9-8b61-0dc25e694b5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:16:01.659779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.659626 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:01.662167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.662078 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w" pod "0609dd55-fe7f-49d9-8b61-0dc25e694b5d" (UID: "0609dd55-fe7f-49d9-8b61-0dc25e694b5d"). InnerVolumeSpecName "kube-api-access-rpp9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:16:01.662644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.662596 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls" pod "0609dd55-fe7f-49d9-8b61-0dc25e694b5d" (UID: "0609dd55-fe7f-49d9-8b61-0dc25e694b5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:16:01.670469 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.670435 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs" pod "0609dd55-fe7f-49d9-8b61-0dc25e694b5d" (UID: "0609dd55-fe7f-49d9-8b61-0dc25e694b5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:16:01.761608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.761570 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:01.761608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.761606 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:01.761838 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:01.761623 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpp9w\" (UniqueName: \"kubernetes.io/projected/0609dd55-fe7f-49d9-8b61-0dc25e694b5d-kube-api-access-rpp9w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:02.232852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.232820 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:16:02.233794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.233773 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" containerName="maas-api" Oct 31 23:16:02.233794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.233795 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" containerName="maas-api" Oct 31 23:16:02.233966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.233957 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" containerName="maas-api" Oct 31 23:16:02.238025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.238008 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.244428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.244247 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-8e6a08-serving-cert\"" Oct 31 23:16:02.250636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.250611 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:16:02.381013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.375649 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.381013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.375904 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.381013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.375976 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lzf\" (UniqueName: \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.381013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.376045 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.477952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.477862 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.477952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.477917 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lzf\" (UniqueName: \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.477952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.477947 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.478239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.478122 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.478305 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:16:02.478278 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-serving-cert: secret "maas-api-e2e-shared-a-w4-8e6a08-serving-cert" not found Oct 31 23:16:02.478399 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:16:02.478384 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls podName:7d0ac0ce-89d2-4837-a771-ba4636017709 nodeName:}" failed. No retries permitted until 2026-10-31 23:16:02.978360208 +0000 UTC m=+1540.961102566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls") pod "maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" (UID: "7d0ac0ce-89d2-4837-a771-ba4636017709") : secret "maas-api-e2e-shared-a-w4-8e6a08-serving-cert" not found Oct 31 23:16:02.478624 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.478598 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.480464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.480445 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.491851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.491821 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lzf\" (UniqueName: \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.600615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.600580 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf" Oct 31 23:16:02.626275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.626242 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:16:02.633882 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.633849 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-907d182f-5c7dc69478-r8wzf"] Oct 31 23:16:02.987552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.987514 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:02.990335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:02.990308 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:03.156551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:03.156516 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:03.646810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:03.646777 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:16:03.647258 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:16:03.647233 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0ac0ce_89d2_4837_a771_ba4636017709.slice/crio-7a8d17a47820dfd6612c1037e9db16b0f78636fe94cfd940342b94098cacfd53 WatchSource:0}: Error finding container 7a8d17a47820dfd6612c1037e9db16b0f78636fe94cfd940342b94098cacfd53: Status 404 returned error can't find the container with id 7a8d17a47820dfd6612c1037e9db16b0f78636fe94cfd940342b94098cacfd53 Oct 31 23:16:04.149059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:04.148977 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0609dd55-fe7f-49d9-8b61-0dc25e694b5d" path="/var/lib/kubelet/pods/0609dd55-fe7f-49d9-8b61-0dc25e694b5d/volumes" Oct 31 23:16:04.688055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:04.687114 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" event={"ID":"7d0ac0ce-89d2-4837-a771-ba4636017709","Type":"ContainerStarted","Data":"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176"} Oct 31 23:16:04.688055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:04.687290 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" event={"ID":"7d0ac0ce-89d2-4837-a771-ba4636017709","Type":"ContainerStarted","Data":"7a8d17a47820dfd6612c1037e9db16b0f78636fe94cfd940342b94098cacfd53"} Oct 31 23:16:04.688586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:04.688532 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:04.742619 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:04.742564 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" podStartSLOduration=2.283935653 podStartE2EDuration="2.742547954s" totalImagesPullingTime="458.612301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:04.738243503 +0000 UTC m=+1542.720985894" watchObservedRunningTime="2026-10-31 23:16:04.742547954 +0000 UTC m=+1542.725290389" Oct 31 23:16:11.596606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:11.596574 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:16:11.740461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:11.740431 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:16:14.891801 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:16:14.891755 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:16:14.938574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:14.938539 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:16:14.943775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:14.943746 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:14.951130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:14.951059 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-f80dfa-serving-cert\"" Oct 31 23:16:14.957145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:14.957116 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:16:15.004429 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.004372 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6dp\" (UniqueName: \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.004601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.004575 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.004653 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.004633 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.004795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.004781 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.106605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.106567 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.106800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.106735 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6dp\" (UniqueName: \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.106850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.106829 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.106883 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.106858 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.107548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.107523 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.109205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.109182 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.109322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.109204 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.119878 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.119853 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6dp\" (UniqueName: \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\") pod \"maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.275882 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.275843 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:15.570571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.570540 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:16:15.570941 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:16:15.570909 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8ce167_55ad_47ac_9cee_731114a42433.slice/crio-d71bc4187b5240ad01884323685e80c1533c77ffb50b1c7621a4e5500a6245f3 WatchSource:0}: Error finding container d71bc4187b5240ad01884323685e80c1533c77ffb50b1c7621a4e5500a6245f3: Status 404 returned error can't find the container with id d71bc4187b5240ad01884323685e80c1533c77ffb50b1c7621a4e5500a6245f3 Oct 31 23:16:15.889661 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:15.889489 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" event={"ID":"2a8ce167-55ad-47ac-9cee-731114a42433","Type":"ContainerStarted","Data":"d71bc4187b5240ad01884323685e80c1533c77ffb50b1c7621a4e5500a6245f3"} Oct 31 23:16:16.898663 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:16.898617 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" event={"ID":"2a8ce167-55ad-47ac-9cee-731114a42433","Type":"ContainerStarted","Data":"598f0188877aceb8556f10421f813648003e225109be621b747c224cf948ff84"} Oct 31 23:16:16.899179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:16.898699 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:16.921960 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:16.921902 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" podStartSLOduration=2.476110507 podStartE2EDuration="2.92188603s" totalImagesPullingTime="445.775523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:16.921057742 +0000 UTC m=+1554.903800122" watchObservedRunningTime="2026-10-31 23:16:16.92188603 +0000 UTC m=+1554.904628448" Oct 31 23:16:22.958520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:22.958490 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:16:33.740462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.740421 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:16:33.750425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.750392 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:33.754431 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.754407 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:16:33.754869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.754826 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d6685e-serving-cert\"" Oct 31 23:16:33.907805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.907344 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:33.907805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.907693 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96lq\" (UniqueName: \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:33.908116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.907827 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:33.908116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:33.907862 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.009140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.009107 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96lq\" (UniqueName: \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.009355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.009176 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.009355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.009195 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.009487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.009358 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.010065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.010014 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.011693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.011663 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.011842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.011820 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.021973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.021948 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96lq\" (UniqueName: \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\") pod \"maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.070024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.069973 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:34.367586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:34.367557 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:16:34.368076 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:16:34.368042 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592e575f_eeea_4d6e_bf00_8e258d9f7929.slice/crio-9834f1b949faf27435eb2851b871aeaacac818faa14ee5f51f9fb7a6793ec620 WatchSource:0}: Error finding container 9834f1b949faf27435eb2851b871aeaacac818faa14ee5f51f9fb7a6793ec620: Status 404 returned error can't find the container with id 9834f1b949faf27435eb2851b871aeaacac818faa14ee5f51f9fb7a6793ec620 Oct 31 23:16:35.176543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:35.176457 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" event={"ID":"592e575f-eeea-4d6e-bf00-8e258d9f7929","Type":"ContainerStarted","Data":"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6"} Oct 31 23:16:35.176543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:35.176534 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" event={"ID":"592e575f-eeea-4d6e-bf00-8e258d9f7929","Type":"ContainerStarted","Data":"9834f1b949faf27435eb2851b871aeaacac818faa14ee5f51f9fb7a6793ec620"} Oct 31 23:16:35.177857 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:35.177831 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:35.202487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:35.202423 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" podStartSLOduration=1.68988904 podStartE2EDuration="2.202404941s" totalImagesPullingTime="512.515901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:35.198095149 +0000 UTC m=+1573.180837529" watchObservedRunningTime="2026-10-31 23:16:35.202404941 +0000 UTC m=+1573.185147321" Oct 31 23:16:37.552592 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.552559 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:16:37.558615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.558587 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.588507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.577611 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:16:37.650358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.650327 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8e6a08-openshift-default-dockercfg-btdj9\"" Oct 31 23:16:37.697704 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.697670 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.697900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.697813 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.697968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.697937 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698029 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698164 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698209 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698335 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698355 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.698467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.698393 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwf4\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.799708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.799673 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.799708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.799711 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.799735 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwf4\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.799846 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.799937 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800019 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800062 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800125 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800246 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800499 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800563 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.800849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.800825 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.801043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.801020 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.801352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.801328 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.803235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.803172 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.803654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.803636 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.812871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.812848 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.812957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.812884 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwf4\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:37.896253 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:37.896218 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:38.196744 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:16:38.196712 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722083a0_8d6b_40e7_be1d_011aaa3df44e.slice/crio-f57a6fc0bfb72c15470114e98a9c04cfdd38dc707380eed79dd94489d80b008e WatchSource:0}: Error finding container f57a6fc0bfb72c15470114e98a9c04cfdd38dc707380eed79dd94489d80b008e: Status 404 returned error can't find the container with id f57a6fc0bfb72c15470114e98a9c04cfdd38dc707380eed79dd94489d80b008e Oct 31 23:16:38.197067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:38.197043 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:16:38.199586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:38.199549 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:16:38.199697 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:38.199627 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:16:38.199697 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:38.199657 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:16:39.211202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.211165 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" event={"ID":"722083a0-8d6b-40e7-be1d-011aaa3df44e","Type":"ContainerStarted","Data":"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b"} Oct 31 23:16:39.211607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.211230 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" event={"ID":"722083a0-8d6b-40e7-be1d-011aaa3df44e","Type":"ContainerStarted","Data":"f57a6fc0bfb72c15470114e98a9c04cfdd38dc707380eed79dd94489d80b008e"} Oct 31 23:16:39.271932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.271861 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podStartSLOduration=2.271839314 podStartE2EDuration="2.271839314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:39.241322936 +0000 UTC m=+1577.224065316" watchObservedRunningTime="2026-10-31 23:16:39.271839314 +0000 UTC m=+1577.254581694" Oct 31 23:16:39.897411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.897376 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:39.899435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.899410 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:16:39.899579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:39.899472 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 23:16:40.912011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:40.911548 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:16:40.912011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:40.911627 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 23:16:41.896802 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:41.896762 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:16:41.897007 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:41.896823 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 23:16:42.198898 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:42.198819 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:16:42.896787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:42.896745 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:16:42.896978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:42.896814 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 23:16:43.905012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:43.899372 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:16:43.905012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:43.899431 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 31 23:16:44.893665 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:16:44.893626 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:16:44.941799 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:44.940687 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:44.941799 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:44.941728 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:44.942567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:44.942544 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:16:47.616873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.616832 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:16:47.622130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.622099 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.635105 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.635071 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-7c4c91-serving-cert\"" Oct 31 23:16:47.638687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.637979 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:16:47.706248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.706204 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.706459 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.706264 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhmj\" (UniqueName: \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.706459 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.706291 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.706635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.706603 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.808589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.808547 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.808780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.808759 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.808877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.808784 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhmj\" (UniqueName: \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.808877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.808810 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.809550 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.809524 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.812532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.812499 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.812636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.812508 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.822699 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.822671 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhmj\" (UniqueName: \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\") pod \"maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:47.953947 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:47.953869 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:48.422456 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:48.422425 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:16:49.401862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:49.401822 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" event={"ID":"374a01e6-c99b-48f0-8fcc-466bf1ea96cb","Type":"ContainerStarted","Data":"1eb272bf75b76fc4be0a08c4b72bfc19bd1cf9d3c4251c314ca1445f25705b24"} Oct 31 23:16:49.402287 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:49.401894 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:16:49.402287 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:49.401916 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" event={"ID":"374a01e6-c99b-48f0-8fcc-466bf1ea96cb","Type":"ContainerStarted","Data":"78b1a9eb784f889fd0bb51dfa795193ccadfd4dd8de542487ad34b879bc825f5"} Oct 31 23:16:49.425614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:49.425539 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" podStartSLOduration=1.941059794 podStartE2EDuration="2.42551892s" totalImagesPullingTime="484.459126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:49.422483948 +0000 UTC m=+1587.405226327" watchObservedRunningTime="2026-10-31 23:16:49.42551892 +0000 UTC m=+1587.408261300" Oct 31 23:16:55.424483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:16:55.424448 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:17:01.343316 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.343280 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:17:01.349852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.349828 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.355770 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.354924 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-36a15fc5-serving-cert\"" Oct 31 23:17:01.358844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.358817 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:17:01.488530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.488490 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.488700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.488679 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.488761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.488747 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbj9r\" (UniqueName: \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.488800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.488783 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.601070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.600261 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.601070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.600374 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbj9r\" (UniqueName: \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.601070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.600430 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.601070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.600820 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.602446 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.602417 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.617698 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.610098 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.622117 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.622081 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbj9r\" (UniqueName: \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.625462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.623161 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\") pod \"maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:01.670402 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:01.670366 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:02.023348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:02.022500 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:17:02.039650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:02.039620 7074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:17:02.793299 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:02.784159 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" event={"ID":"ab793972-6ecb-4f23-bf0c-15407489fa80","Type":"ContainerStarted","Data":"7e77308432a2063c668e7e9aa31a28bb2065a63ad5c8003a3bad7a8cb1a93adf"} Oct 31 23:17:03.793694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:03.793656 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" event={"ID":"ab793972-6ecb-4f23-bf0c-15407489fa80","Type":"ContainerStarted","Data":"a6c3f8a9dd232fc2c5d1c8b1174914f5e4a9cabdd6bde71acac11634e2d4a4c0"} Oct 31 23:17:03.794231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:03.793740 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:03.817972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:03.817924 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" podStartSLOduration=2.39946869 podStartE2EDuration="2.817910089s" totalImagesPullingTime="418.441399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:03.81489885 +0000 UTC m=+1601.797641229" watchObservedRunningTime="2026-10-31 23:17:03.817910089 +0000 UTC m=+1601.800652467" Oct 31 23:17:07.645066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.642980 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:17:07.650955 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.650929 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.683723 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.683669 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:17:07.783395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.781656 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.783997 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.783945 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.784373 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.784255 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.784576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.784557 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.784868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.784846 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.785154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.785133 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzt9\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.785400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.785381 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.785574 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.785558 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.785748 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.785732 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.922554 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934262 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.933832 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934477 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934518 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934667 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934786 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzt9\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934889 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.934949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934920 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.935144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.934954 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.935562 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.935531 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.936175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.936143 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.937086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.936535 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.937252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.936799 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.971161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.970801 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.986214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.986179 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzt9\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:07.991074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:07.991004 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:08.029013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.025819 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:08.282928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.282891 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:08.590835 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.590802 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:17:08.590979 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:17:08.590813 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a55d87_5ecd_4ffb_a601_701e66281d43.slice/crio-2ae14dec16d738e8c0588e5cfd42182a5de13a41a7dd1ff4ff0aade053534bd5 WatchSource:0}: Error finding container 2ae14dec16d738e8c0588e5cfd42182a5de13a41a7dd1ff4ff0aade053534bd5: Status 404 returned error can't find the container with id 2ae14dec16d738e8c0588e5cfd42182a5de13a41a7dd1ff4ff0aade053534bd5 Oct 31 23:17:08.595551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.595511 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:17:08.595654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.595587 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:17:08.595654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.595629 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:17:08.998512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.998476 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" event={"ID":"47a55d87-5ecd-4ffb-a601-701e66281d43","Type":"ContainerStarted","Data":"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392"} Oct 31 23:17:08.998918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:08.998535 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" event={"ID":"47a55d87-5ecd-4ffb-a601-701e66281d43","Type":"ContainerStarted","Data":"2ae14dec16d738e8c0588e5cfd42182a5de13a41a7dd1ff4ff0aade053534bd5"} Oct 31 23:17:09.036671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:09.036603 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podStartSLOduration=2.036583081 podStartE2EDuration="2.036583081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:09.028424365 +0000 UTC m=+1607.011166756" watchObservedRunningTime="2026-10-31 23:17:09.036583081 +0000 UTC m=+1607.019325459" Oct 31 23:17:09.283578 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:09.283476 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:09.289902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:09.286406 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:09.289902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:09.286475 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:09.886082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:09.886048 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:17:10.283854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:10.283811 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:10.284310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:10.283893 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:11.306929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:11.306890 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:11.307606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:11.307557 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:12.284841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:12.284730 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:12.285327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:12.285277 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:13.288213 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:13.288161 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:13.288599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:13.288239 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:14.284145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:14.284104 7074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:17:14.284337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:14.284168 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 23:17:14.905685 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:17:14.905642 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:17:15.401527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:15.399423 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:15.401527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:15.401342 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:15.404394 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:15.404367 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:17:20.719089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.719053 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:17:20.724401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.724375 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.731676 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.731645 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c4d32e-serving-cert\"" Oct 31 23:17:20.736427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.736399 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:17:20.810815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.810773 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprhl\" (UniqueName: \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.810815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.810821 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.811093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.810983 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.811203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.811181 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.917446 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.917405 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.940012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.925338 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprhl\" (UniqueName: \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.940012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.925394 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.940012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.925552 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.940012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.929077 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.940734 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.940703 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.961650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.961621 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:20.972773 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:20.972662 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprhl\" (UniqueName: \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\") pod \"maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:21.045504 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:21.045466 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:21.365216 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:17:21.365184 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d42afb_923e_490b_b388_d2a935f8f12a.slice/crio-a919986a1b5fd4ed6eaf174b14c704ea7811a61e98e78b4494f3ccb5ea293fa9 WatchSource:0}: Error finding container a919986a1b5fd4ed6eaf174b14c704ea7811a61e98e78b4494f3ccb5ea293fa9: Status 404 returned error can't find the container with id a919986a1b5fd4ed6eaf174b14c704ea7811a61e98e78b4494f3ccb5ea293fa9 Oct 31 23:17:21.365319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:21.365260 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:17:22.176452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:22.176371 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" event={"ID":"e6d42afb-923e-490b-b388-d2a935f8f12a","Type":"ContainerStarted","Data":"89f6145d8d7faa9f5bf602daa844fcddfaaa2810f0a82f8d3427e84011bf62e4"} Oct 31 23:17:22.176452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:22.176423 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" event={"ID":"e6d42afb-923e-490b-b388-d2a935f8f12a","Type":"ContainerStarted","Data":"a919986a1b5fd4ed6eaf174b14c704ea7811a61e98e78b4494f3ccb5ea293fa9"} Oct 31 23:17:22.176867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:22.176567 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:22.199031 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:22.198963 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" podStartSLOduration=1.660121387 podStartE2EDuration="2.198949118s" totalImagesPullingTime="538.827731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:22.197365692 +0000 UTC m=+1620.180108071" watchObservedRunningTime="2026-10-31 23:17:22.198949118 +0000 UTC m=+1620.181691495" Oct 31 23:17:28.210375 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:28.210341 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:17:44.891613 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:17:44.891563 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:17:49.816702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.816662 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:17:49.823084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.823054 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.832549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.832523 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d9e90c-serving-cert\"" Oct 31 23:17:49.837028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.836984 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:17:49.870380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.870346 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmlv\" (UniqueName: \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.870576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.870539 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.870668 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.870652 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.870813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.870780 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.978939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.978572 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.979693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.979662 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.980130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.980104 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.980798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.980513 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmlv\" (UniqueName: \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.981019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.980955 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.983544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.983519 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:49.983708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:49.983687 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:50.026269 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:50.026234 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmlv\" (UniqueName: \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\") pod \"maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:50.150415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:50.150324 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:50.452172 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:50.452132 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:17:50.455819 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:17:50.455790 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183fdfd_22de_4bdb_9b13_f3bdc1bbe314.slice/crio-65bdcb3d10f34238b4ec085a2e6f7579038c55d8f5659c3090f83255eb59cbc1 WatchSource:0}: Error finding container 65bdcb3d10f34238b4ec085a2e6f7579038c55d8f5659c3090f83255eb59cbc1: Status 404 returned error can't find the container with id 65bdcb3d10f34238b4ec085a2e6f7579038c55d8f5659c3090f83255eb59cbc1 Oct 31 23:17:50.533015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:50.525760 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" event={"ID":"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314","Type":"ContainerStarted","Data":"65bdcb3d10f34238b4ec085a2e6f7579038c55d8f5659c3090f83255eb59cbc1"} Oct 31 23:17:51.555275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:51.555073 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" event={"ID":"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314","Type":"ContainerStarted","Data":"5773b9a4c0e3dea4a6c09a90c84f497e28b6b4a908181a25f61b8696df1aae30"} Oct 31 23:17:51.566453 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:51.557097 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:58.580447 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:58.580415 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:17:58.602747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:17:58.602682 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" podStartSLOduration=9.139286012 podStartE2EDuration="9.602667659s" totalImagesPullingTime="463.381647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:51.595322586 +0000 UTC m=+1649.578065032" watchObservedRunningTime="2026-10-31 23:17:58.602667659 +0000 UTC m=+1656.585410036" Oct 31 23:18:14.897892 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:14.897847 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:18:20.795882 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.795846 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:18:20.809161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.809130 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.819339 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.819304 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:18:20.840070 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.840031 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2126ff-serving-cert\"" Oct 31 23:18:20.867467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.867430 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.867868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.867838 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9hc\" (UniqueName: \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.868158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.868133 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.868345 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.868327 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.990142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.986375 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.990142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.986459 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9hc\" (UniqueName: \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.990142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.986593 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.990142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.986652 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:20.990142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:20.987904 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:21.005163 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:21.005126 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-serving-cert: secret "maas-api-e2e-shared-a-w3-2126ff-serving-cert" not found Oct 31 23:18:21.005482 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:21.005452 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls podName:3fc753cb-a030-47bc-a64d-dee62ff222b0 nodeName:}" failed. No retries permitted until 2026-10-31 23:18:21.505419529 +0000 UTC m=+1679.488161929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" (UID: "3fc753cb-a030-47bc-a64d-dee62ff222b0") : secret "maas-api-e2e-shared-a-w3-2126ff-serving-cert" not found Oct 31 23:18:21.077099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:21.076947 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:21.133362 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:21.125959 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9hc\" (UniqueName: \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:21.598389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:21.598353 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:21.600885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:21.600860 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:21.817444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:21.817406 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:22.205130 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:22.205101 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:18:22.205807 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:18:22.205778 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc753cb_a030_47bc_a64d_dee62ff222b0.slice/crio-424b1177ec9c7a21129ea4e4d8214b64bc1a28f47f2b3711941e76aec85486ac WatchSource:0}: Error finding container 424b1177ec9c7a21129ea4e4d8214b64bc1a28f47f2b3711941e76aec85486ac: Status 404 returned error can't find the container with id 424b1177ec9c7a21129ea4e4d8214b64bc1a28f47f2b3711941e76aec85486ac Oct 31 23:18:23.160324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:23.160280 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" event={"ID":"3fc753cb-a030-47bc-a64d-dee62ff222b0","Type":"ContainerStarted","Data":"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54"} Oct 31 23:18:23.160950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:23.160345 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" event={"ID":"3fc753cb-a030-47bc-a64d-dee62ff222b0","Type":"ContainerStarted","Data":"424b1177ec9c7a21129ea4e4d8214b64bc1a28f47f2b3711941e76aec85486ac"} Oct 31 23:18:23.160950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:23.160522 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:23.197956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:23.197898 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" podStartSLOduration=2.700891713 podStartE2EDuration="3.197879522s" totalImagesPullingTime="496.987809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:23.195258136 +0000 UTC m=+1681.178000515" watchObservedRunningTime="2026-10-31 23:18:23.197879522 +0000 UTC m=+1681.180621903" Oct 31 23:18:28.720448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.720414 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:18:28.728326 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.727814 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:28.740439 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.740406 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85b90bac-serving-cert\"" Oct 31 23:18:28.745094 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.744909 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:18:28.889154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.889060 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7q2k\" (UniqueName: \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:28.896090 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.896053 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:28.896623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.896582 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:28.896921 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:28.896901 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.000275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.000250 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7q2k\" (UniqueName: \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.000454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.000352 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.000522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.000456 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.000584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.000519 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.001252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.001227 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.002856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.002833 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.002964 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.002863 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.013140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.013116 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7q2k\" (UniqueName: \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\") pod \"maas-api-e2e-mt-85b90bac-fd899b454-7lpbn\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.060854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.060816 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:29.175231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.175204 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:18:29.428909 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:29.428827 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:18:30.244251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:30.244161 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" event={"ID":"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1","Type":"ContainerStarted","Data":"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb"} Oct 31 23:18:30.244251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:30.244218 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" event={"ID":"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1","Type":"ContainerStarted","Data":"4be19d8d5fce54594d056a8a1ffc9936a510bfc723450f7bfa7abcea27ba406c"} Oct 31 23:18:30.244251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:30.244249 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:30.267722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:30.267677 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" podStartSLOduration=1.728627484 podStartE2EDuration="2.26766206s" totalImagesPullingTime="539.034576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:30.264924781 +0000 UTC m=+1688.247667160" watchObservedRunningTime="2026-10-31 23:18:30.26766206 +0000 UTC m=+1688.250404435" Oct 31 23:18:36.260606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:36.260579 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:18:44.892780 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:44.892740 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:18:57.228176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.228138 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:18:57.251250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.251218 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:18:57.251435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.251337 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.256032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.255978 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-270e8c-serving-cert\"" Oct 31 23:18:57.258140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.258111 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7kh\" (UniqueName: \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.258566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.258545 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.258727 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.258707 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.258969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.258949 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.383011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.382894 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.383011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.382974 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.383260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.383119 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.383260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.383195 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7kh\" (UniqueName: \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.386649 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:57.385306 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-serving-cert: secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:18:57.386649 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:57.385436 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls podName:89021da0-1ab9-4a7c-bda0-4b465e3b801b nodeName:}" failed. No retries permitted until 2026-10-31 23:18:57.885400684 +0000 UTC m=+1715.868143043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b") : secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:18:57.387595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.387548 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.389388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.389359 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.401349 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.401319 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7kh\" (UniqueName: \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.898774 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:57.898745 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:57.899154 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:57.899066 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-serving-cert: secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:18:57.899318 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:57.899305 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls podName:89021da0-1ab9-4a7c-bda0-4b465e3b801b nodeName:}" failed. No retries permitted until 2026-10-31 23:18:58.899281359 +0000 UTC m=+1716.882023733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b") : secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:18:58.920011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:18:58.917038 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:18:58.920011 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:58.917651 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-serving-cert: secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:18:58.920011 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:18:58.917737 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls podName:89021da0-1ab9-4a7c-bda0-4b465e3b801b nodeName:}" failed. No retries permitted until 2026-10-31 23:19:00.917717441 +0000 UTC m=+1718.900459814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b") : secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:19:00.963966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:00.963931 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:00.964442 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:19:00.964113 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-serving-cert: secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:19:00.964442 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:19:00.964185 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls podName:89021da0-1ab9-4a7c-bda0-4b465e3b801b nodeName:}" failed. No retries permitted until 2026-10-31 23:19:04.964167451 +0000 UTC m=+1722.946909830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b") : secret "maas-api-e2e-shared-b-w3-270e8c-serving-cert" not found Oct 31 23:19:05.067452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:05.067240 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:05.071773 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:05.071739 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:05.371100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:05.371017 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:05.695173 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:05.695146 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:19:05.698585 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:19:05.698559 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89021da0_1ab9_4a7c_bda0_4b465e3b801b.slice/crio-d6923adc22a68b4fd02b04ce8d9bebad7f096c07059221856403d195a8d16609 WatchSource:0}: Error finding container d6923adc22a68b4fd02b04ce8d9bebad7f096c07059221856403d195a8d16609: Status 404 returned error can't find the container with id d6923adc22a68b4fd02b04ce8d9bebad7f096c07059221856403d195a8d16609 Oct 31 23:19:05.805076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:05.805044 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" event={"ID":"89021da0-1ab9-4a7c-bda0-4b465e3b801b","Type":"ContainerStarted","Data":"d6923adc22a68b4fd02b04ce8d9bebad7f096c07059221856403d195a8d16609"} Oct 31 23:19:06.817922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:06.817879 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" event={"ID":"89021da0-1ab9-4a7c-bda0-4b465e3b801b","Type":"ContainerStarted","Data":"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af"} Oct 31 23:19:06.818341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:06.818080 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:06.845433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:06.845379 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" podStartSLOduration=9.39458679 podStartE2EDuration="9.84536482s" totalImagesPullingTime="450.77803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:19:06.841240726 +0000 UTC m=+1724.823983103" watchObservedRunningTime="2026-10-31 23:19:06.84536482 +0000 UTC m=+1724.828107198" Oct 31 23:19:12.832149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:19:12.832117 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:19:14.892784 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:19:14.892736 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:19:44.892688 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:19:44.892653 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:20:14.891113 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:14.891068 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:20:22.464718 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:22.464690 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:20:22.468022 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:22.467956 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:20:39.934376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:39.934327 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:20:39.941111 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:39.941081 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:39.948036 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:39.947973 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-479107-serving-cert\"" Oct 31 23:20:39.956053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:39.954241 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:20:40.082456 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.061543 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.102277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.102240 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.120839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.120800 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.121267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.121243 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62m4c\" (UniqueName: \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.223892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.223800 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62m4c\" (UniqueName: \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.224183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.224164 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.224267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.224211 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.224400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.224377 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.224544 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:40.224520 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-479107-serving-cert: secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:40.224617 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:40.224604 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls podName:d917e2f7-2c99-4abe-bba1-083cb4c69e4f nodeName:}" failed. No retries permitted until 2026-10-31 23:20:40.724586123 +0000 UTC m=+1818.707328482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls") pod "maas-api-e2e-api-a-479107-555b99b64c-ww2mq" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f") : secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:40.224891 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.224870 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.226746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.226728 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.236194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.236171 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62m4c\" (UniqueName: \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.508332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.502112 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:20:40.508332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.502450 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" podUID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" containerName="maas-api" containerID="cri-o://dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb" gracePeriod=30 Oct 31 23:20:40.742016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:40.734653 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:40.742016 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:40.735081 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-479107-serving-cert: secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:40.742016 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:40.737038 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls podName:d917e2f7-2c99-4abe-bba1-083cb4c69e4f nodeName:}" failed. No retries permitted until 2026-10-31 23:20:41.737010336 +0000 UTC m=+1819.719752709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls") pod "maas-api-e2e-api-a-479107-555b99b64c-ww2mq" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f") : secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:41.040598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.040485 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:20:41.050973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.050935 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.063502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.063451 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4528ed-serving-cert\"" Oct 31 23:20:41.071072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.071046 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:20:41.153102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.153047 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gvf\" (UniqueName: \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.153296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.153204 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.153296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.153273 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.153761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.153562 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.231257 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.228456 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.255207 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gvf\" (UniqueName: \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.255375 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.255442 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.255554 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.258697 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-serving-cert: secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:41.259071 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.258760 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls podName:438d0ae1-b63e-47a7-bfa3-39e9c66c3a88 nodeName:}" failed. No retries permitted until 2026-10-31 23:20:41.758740112 +0000 UTC m=+1819.741482481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls") pod "maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88") : secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:41.264709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.259771 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.282068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.270628 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.295058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.290215 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gvf\" (UniqueName: \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.303630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.300943 7074 generic.go:356] "Generic (PLEG): container finished" podID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" containerID="dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb" exitCode=0 Oct 31 23:20:41.303630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.301173 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" event={"ID":"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1","Type":"ContainerDied","Data":"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb"} Oct 31 23:20:41.303630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.301196 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" Oct 31 23:20:41.303630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.301256 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn" event={"ID":"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1","Type":"ContainerDied","Data":"4be19d8d5fce54594d056a8a1ffc9936a510bfc723450f7bfa7abcea27ba406c"} Oct 31 23:20:41.303630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.301298 7074 scope.go:122] "RemoveContainer" containerID="dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb" Oct 31 23:20:41.332357 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.332324 7074 scope.go:122] "RemoveContainer" containerID="dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb" Oct 31 23:20:41.332823 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.332796 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb\": container with ID starting with dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb not found: ID does not exist" containerID="dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb" Oct 31 23:20:41.332954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.332930 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb"} err="failed to get container status \"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb\": rpc error: code = NotFound desc = could not find container \"dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb\": container with ID starting with dc5406f3cc2536008c652048eb363ae65aa26c7969b7a5e59d4763bbbd2515eb not found: ID does not exist" Oct 31 23:20:41.358045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.357270 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\") pod \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " Oct 31 23:20:41.358045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.357386 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\") pod \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " Oct 31 23:20:41.358045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.357541 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\" (UniqueName: \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\") pod \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " Oct 31 23:20:41.358045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.357707 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\") pod \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\" (UID: \"f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1\") " Oct 31 23:20:41.359237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.359202 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca" pod "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" (UID: "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:20:41.360570 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.360537 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k" pod "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" (UID: "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1"). InnerVolumeSpecName "kube-api-access-q7q2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:20:41.364050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.363049 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs" pod "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" (UID: "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:20:41.376501 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.376472 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls" pod "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" (UID: "f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:20:41.460543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.460504 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:20:41.460543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.460542 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:20:41.460739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.460554 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7q2k\" (UniqueName: \"kubernetes.io/projected/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-kube-api-access-q7q2k\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:20:41.460739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.460564 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:20:41.659079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.659031 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:20:41.677017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.666097 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85b90bac-fd899b454-7lpbn"] Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.776314 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:41.776407 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.776782 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-479107-serving-cert: secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.776874 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls podName:d917e2f7-2c99-4abe-bba1-083cb4c69e4f nodeName:}" failed. No retries permitted until 2026-10-31 23:20:43.776851458 +0000 UTC m=+1821.759593857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls") pod "maas-api-e2e-api-a-479107-555b99b64c-ww2mq" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f") : secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.777384 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-serving-cert: secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:41.780017 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:41.777436 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls podName:438d0ae1-b63e-47a7-bfa3-39e9c66c3a88 nodeName:}" failed. No retries permitted until 2026-10-31 23:20:42.777419886 +0000 UTC m=+1820.760162261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls") pod "maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88") : secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:42.169120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:42.169080 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" path="/var/lib/kubelet/pods/f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1/volumes" Oct 31 23:20:42.795363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:42.795329 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:42.795552 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:42.795486 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-serving-cert: secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:42.795552 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:42.795550 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls podName:438d0ae1-b63e-47a7-bfa3-39e9c66c3a88 nodeName:}" failed. No retries permitted until 2026-10-31 23:20:44.795534675 +0000 UTC m=+1822.778277044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls") pod "maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88") : secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:43.834604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:43.834566 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:43.835134 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:43.834752 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-479107-serving-cert: secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:43.835134 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:43.834839 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls podName:d917e2f7-2c99-4abe-bba1-083cb4c69e4f nodeName:}" failed. No retries permitted until 2026-10-31 23:20:47.834816998 +0000 UTC m=+1825.817559369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls") pod "maas-api-e2e-api-a-479107-555b99b64c-ww2mq" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f") : secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:44.861494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:44.861455 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:44.861978 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:44.861577 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-serving-cert: secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:44.861978 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:44.861651 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls podName:438d0ae1-b63e-47a7-bfa3-39e9c66c3a88 nodeName:}" failed. No retries permitted until 2026-10-31 23:20:48.861634517 +0000 UTC m=+1826.844376896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls") pod "maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88") : secret "maas-api-e2e-api-a-4528ed-serving-cert" not found Oct 31 23:20:44.890516 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:44.890480 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:20:47.924370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:47.924334 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:47.924811 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:47.924485 7074 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-479107-serving-cert: secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:47.924811 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:20:47.924554 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls podName:d917e2f7-2c99-4abe-bba1-083cb4c69e4f nodeName:}" failed. No retries permitted until 2026-10-31 23:20:55.924539896 +0000 UTC m=+1833.907282273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls") pod "maas-api-e2e-api-a-479107-555b99b64c-ww2mq" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f") : secret "maas-api-e2e-api-a-479107-serving-cert" not found Oct 31 23:20:48.876011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:48.875110 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:48.910011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:48.901856 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:49.202063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:49.201939 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:49.504454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:49.504425 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:20:49.504654 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:20:49.504627 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438d0ae1_b63e_47a7_bfa3_39e9c66c3a88.slice/crio-3c1698bc5d17a936224a1fe02406dc548445adfd769970d0e4eb05a9db8a061e WatchSource:0}: Error finding container 3c1698bc5d17a936224a1fe02406dc548445adfd769970d0e4eb05a9db8a061e: Status 404 returned error can't find the container with id 3c1698bc5d17a936224a1fe02406dc548445adfd769970d0e4eb05a9db8a061e Oct 31 23:20:50.488009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:50.487860 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" event={"ID":"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88","Type":"ContainerStarted","Data":"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405"} Oct 31 23:20:50.488009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:50.487926 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" event={"ID":"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88","Type":"ContainerStarted","Data":"3c1698bc5d17a936224a1fe02406dc548445adfd769970d0e4eb05a9db8a061e"} Oct 31 23:20:50.500029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:50.489279 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:50.521040 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:50.510624 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" podStartSLOduration=9.097450664 podStartE2EDuration="9.510609522s" totalImagesPullingTime="413.158858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:50.510053485 +0000 UTC m=+1828.492795861" watchObservedRunningTime="2026-10-31 23:20:50.510609522 +0000 UTC m=+1828.493351955" Oct 31 23:20:55.957080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:55.957039 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:55.959480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:55.959457 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"maas-api-e2e-api-a-479107-555b99b64c-ww2mq\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:56.164762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:56.164725 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:20:56.458140 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:20:56.458107 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd917e2f7_2c99_4abe_bba1_083cb4c69e4f.slice/crio-0091cdbe526bb1e90c380f3de6691e69460e27bc63f966b069968c69760796ec WatchSource:0}: Error finding container 0091cdbe526bb1e90c380f3de6691e69460e27bc63f966b069968c69760796ec: Status 404 returned error can't find the container with id 0091cdbe526bb1e90c380f3de6691e69460e27bc63f966b069968c69760796ec Oct 31 23:20:56.461326 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:56.460490 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:20:56.546266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:56.546229 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" event={"ID":"d917e2f7-2c99-4abe-bba1-083cb4c69e4f","Type":"ContainerStarted","Data":"0091cdbe526bb1e90c380f3de6691e69460e27bc63f966b069968c69760796ec"} Oct 31 23:20:57.532841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:57.532808 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:20:57.568475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:57.568434 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" event={"ID":"d917e2f7-2c99-4abe-bba1-083cb4c69e4f","Type":"ContainerStarted","Data":"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a"} Oct 31 23:20:57.569758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:20:57.569731 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:21:04.590736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:04.590707 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:21:04.616544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:04.616483 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" podStartSLOduration=25.175167649 podStartE2EDuration="25.616467718s" totalImagesPullingTime="441.300069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:20:57.595604505 +0000 UTC m=+1835.578346881" watchObservedRunningTime="2026-10-31 23:21:04.616467718 +0000 UTC m=+1842.599210121" Oct 31 23:21:15.055053 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:21:15.053857 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:21:15.285888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:15.285851 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:21:15.323840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:15.323753 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:21:20.286444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:20.286392 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" containerID="cri-o://7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b" gracePeriod=25 Oct 31 23:21:20.324674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:20.324626 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" containerID="cri-o://d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442" gracePeriod=25 Oct 31 23:21:20.651541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:20.651445 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:21:29.633899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:29.633862 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:21:30.649881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:30.649840 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:21:39.634279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:39.634239 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:21:40.718934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.718907 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:21:40.787002 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.786963 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:21:40.882961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.882925 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\") pod \"514f3f01-5825-420b-93f1-8e8a294ccb70\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " Oct 31 23:21:40.883164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.883096 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\" (UniqueName: \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\") pod \"514f3f01-5825-420b-93f1-8e8a294ccb70\" (UID: \"514f3f01-5825-420b-93f1-8e8a294ccb70\") " Oct 31 23:21:40.883332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.883302 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\") pod \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " Oct 31 23:21:40.883422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.883402 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume" pod "514f3f01-5825-420b-93f1-8e8a294ccb70" (UID: "514f3f01-5825-420b-93f1-8e8a294ccb70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:40.883628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.883533 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\" (UniqueName: \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\") pod \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\" (UID: \"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3\") " Oct 31 23:21:40.883722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.883695 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume" pod "b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" (UID: "b5cf81c8-dfcb-4c44-92fa-a9033644e2c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:40.884464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.884371 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:40.884464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.884400 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514f3f01-5825-420b-93f1-8e8a294ccb70-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:40.886203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.886172 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5" pod "514f3f01-5825-420b-93f1-8e8a294ccb70" (UID: "514f3f01-5825-420b-93f1-8e8a294ccb70"). InnerVolumeSpecName "kube-api-access-wdnf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:40.886337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.886269 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh" pod "b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" (UID: "b5cf81c8-dfcb-4c44-92fa-a9033644e2c3"). InnerVolumeSpecName "kube-api-access-99gzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:40.985967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.985934 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99gzh\" (UniqueName: \"kubernetes.io/projected/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3-kube-api-access-99gzh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:40.985967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:40.985962 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnf5\" (UniqueName: \"kubernetes.io/projected/514f3f01-5825-420b-93f1-8e8a294ccb70-kube-api-access-wdnf5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:41.165927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.165836 7074 generic.go:356] "Generic (PLEG): container finished" podID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerID="7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b" exitCode=0 Oct 31 23:21:41.165927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.165887 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" Oct 31 23:21:41.165927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.165913 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" event={"ID":"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3","Type":"ContainerDied","Data":"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b"} Oct 31 23:21:41.166274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.165964 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2" event={"ID":"b5cf81c8-dfcb-4c44-92fa-a9033644e2c3","Type":"ContainerDied","Data":"7cd1ffb65f7d1896efee70fcd2791cab4de62b5acc5b3179574efdcabcd8b0ec"} Oct 31 23:21:41.166274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.165998 7074 scope.go:122] "RemoveContainer" containerID="7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b" Oct 31 23:21:41.167861 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.167835 7074 generic.go:356] "Generic (PLEG): container finished" podID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerID="d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442" exitCode=0 Oct 31 23:21:41.167959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.167888 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" event={"ID":"514f3f01-5825-420b-93f1-8e8a294ccb70","Type":"ContainerDied","Data":"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442"} Oct 31 23:21:41.167959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.167902 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" Oct 31 23:21:41.167959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.167949 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" event={"ID":"514f3f01-5825-420b-93f1-8e8a294ccb70","Type":"ContainerDied","Data":"727e1ff4c1da5d6d616289a3819fd131afe2f43c1136a8456e59028a8755578b"} Oct 31 23:21:41.180227 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.180127 7074 scope.go:122] "RemoveContainer" containerID="7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b" Oct 31 23:21:41.180497 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:21:41.180471 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b\": container with ID starting with 7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b not found: ID does not exist" containerID="7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b" Oct 31 23:21:41.180560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.180511 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b"} err="failed to get container status \"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b\": rpc error: code = NotFound desc = could not find container \"7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b\": container with ID starting with 7485d43231ccabc789e35a4550216e186f3cb73d74c26b29e40962cd14e4402b not found: ID does not exist" Oct 31 23:21:41.180560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.180536 7074 scope.go:122] "RemoveContainer" containerID="d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442" Oct 31 23:21:41.193077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.193055 7074 scope.go:122] "RemoveContainer" containerID="d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442" Oct 31 23:21:41.193431 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:21:41.193403 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442\": container with ID starting with d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442 not found: ID does not exist" containerID="d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442" Oct 31 23:21:41.193537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.193441 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442"} err="failed to get container status \"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442\": rpc error: code = NotFound desc = could not find container \"d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442\": container with ID starting with d3f408e7eade002de7b50dfcfe0ab703bbf2af4cd5d583bfd04511701cfa0442 not found: ID does not exist" Oct 31 23:21:41.204735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.204711 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:21:41.212377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.212354 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q"] Oct 31 23:21:41.236794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.236753 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:21:41.242556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.242526 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2294a7-846ddcf4f8-krqn2"] Oct 31 23:21:41.650343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:41.650294 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2294a7-7bb58c864c-2gs8q" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.66:50052\" within 1s: context deadline exceeded" Oct 31 23:21:41.650529 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:21:41.650394 7074 logging.go:55] [core] [Channel #3404 SubChannel #3405]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.66:50052", ServerName: "10.134.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.66:50052: operation was canceled" Oct 31 23:21:42.161465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:42.161435 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" path="/var/lib/kubelet/pods/514f3f01-5825-420b-93f1-8e8a294ccb70/volumes" Oct 31 23:21:42.161869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:42.161854 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" path="/var/lib/kubelet/pods/b5cf81c8-dfcb-4c44-92fa-a9033644e2c3/volumes" Oct 31 23:21:44.922039 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:21:44.920028 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:21:50.345266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.345229 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:21:50.345746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.345451 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" podUID="a6a39603-18f1-4383-b261-bbf664a67e61" containerName="maas-api" containerID="cri-o://faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6" gracePeriod=30 Oct 31 23:21:50.924604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.924571 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:21:50.933599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.933568 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\") pod \"a6a39603-18f1-4383-b261-bbf664a67e61\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " Oct 31 23:21:50.933749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.933655 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\") pod \"a6a39603-18f1-4383-b261-bbf664a67e61\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " Oct 31 23:21:50.933749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.933681 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\" (UniqueName: \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\") pod \"a6a39603-18f1-4383-b261-bbf664a67e61\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " Oct 31 23:21:50.933880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.933759 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\") pod \"a6a39603-18f1-4383-b261-bbf664a67e61\" (UID: \"a6a39603-18f1-4383-b261-bbf664a67e61\") " Oct 31 23:21:50.933940 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.933915 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca" pod "a6a39603-18f1-4383-b261-bbf664a67e61" (UID: "a6a39603-18f1-4383-b261-bbf664a67e61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:50.934624 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.934597 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a39603-18f1-4383-b261-bbf664a67e61-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:50.935901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.935879 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls" pod "a6a39603-18f1-4383-b261-bbf664a67e61" (UID: "a6a39603-18f1-4383-b261-bbf664a67e61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:50.936101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.936080 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4" pod "a6a39603-18f1-4383-b261-bbf664a67e61" (UID: "a6a39603-18f1-4383-b261-bbf664a67e61"). InnerVolumeSpecName "kube-api-access-9cdj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:50.938468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:50.938449 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs" pod "a6a39603-18f1-4383-b261-bbf664a67e61" (UID: "a6a39603-18f1-4383-b261-bbf664a67e61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:51.036673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.036636 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:51.036673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.036675 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdj4\" (UniqueName: \"kubernetes.io/projected/a6a39603-18f1-4383-b261-bbf664a67e61-kube-api-access-9cdj4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:51.036903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.036694 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a39603-18f1-4383-b261-bbf664a67e61-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:51.327923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.327853 7074 generic.go:356] "Generic (PLEG): container finished" podID="a6a39603-18f1-4383-b261-bbf664a67e61" containerID="faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6" exitCode=0 Oct 31 23:21:51.327923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.327906 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" Oct 31 23:21:51.328192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.327937 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" event={"ID":"a6a39603-18f1-4383-b261-bbf664a67e61","Type":"ContainerDied","Data":"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6"} Oct 31 23:21:51.328192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.328040 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc" event={"ID":"a6a39603-18f1-4383-b261-bbf664a67e61","Type":"ContainerDied","Data":"534f3b9e83feb6456b1586fd02fde0c9b43f4af8764cf19780a891397a1eaf28"} Oct 31 23:21:51.328192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.328073 7074 scope.go:122] "RemoveContainer" containerID="faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6" Oct 31 23:21:51.343498 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.343018 7074 scope.go:122] "RemoveContainer" containerID="faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6" Oct 31 23:21:51.343498 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:21:51.343327 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6\": container with ID starting with faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6 not found: ID does not exist" containerID="faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6" Oct 31 23:21:51.343498 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.343360 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6"} err="failed to get container status \"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6\": rpc error: code = NotFound desc = could not find container \"faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6\": container with ID starting with faca42db5ad685a9ff7001054cf035db7efe24d30ca44bda9a4b200067a807e6 not found: ID does not exist" Oct 31 23:21:51.373758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.373733 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:21:51.397551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:51.397520 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2294a7-6f8cb6b55c-t99gc"] Oct 31 23:21:52.176448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:21:52.176414 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a39603-18f1-4383-b261-bbf664a67e61" path="/var/lib/kubelet/pods/a6a39603-18f1-4383-b261-bbf664a67e61/volumes" Oct 31 23:22:01.938680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.938628 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:22:01.939237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.938969 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" containerID="cri-o://e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547" gracePeriod=30 Oct 31 23:22:01.948123 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.948086 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:22:01.948392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.948346 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" containerID="cri-o://e2f55d2d1896d21fab87681b86e3a8798e07a9d8d3ca6902a338dc0a04060165" gracePeriod=30 Oct 31 23:22:01.953260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.953233 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:22:01.953492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:01.953470 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" containerID="cri-o://763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06" gracePeriod=30 Oct 31 23:22:02.671069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:02.671027 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:22:02.671257 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:02.671108 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:22:02.832047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:02.831975 7074 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:22:02.832266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:02.832070 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:22:07.512771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.512627 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:22:07.538752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.538720 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:22:07.649064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.649033 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.649334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.649312 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.649546 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.649519 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.649672 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.649651 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.650274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.650193 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:07.650514 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.650495 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.650628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.650607 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.650739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.650722 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.650882 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.650865 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.651032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651015 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.651148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651121 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.651401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651379 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.651532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651491 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:07.651643 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651527 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.651783 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651766 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.651933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651915 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.652092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652074 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.652336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652317 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.652467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652450 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.652640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652622 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.652830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652792 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.653033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.652972 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\") pod \"f1178048-56ee-4374-a2ed-a9478f96705c\" (UID: \"f1178048-56ee-4374-a2ed-a9478f96705c\") " Oct 31 23:22:07.653209 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.653187 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\") pod \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\" (UID: \"82ff1268-d948-4eb5-807b-3e4f9c548f1b\") " Oct 31 23:22:07.654697 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.654673 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.654932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.654916 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.655147 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.655190 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.655206 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1178048-56ee-4374-a2ed-a9478f96705c-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.651820 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.653033 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.653380 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.654861 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.655680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.655119 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.662147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.661930 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:07.666576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.666249 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "kube-api-access-frbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.670685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.670571 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.670685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.670577 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:07.670685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.670635 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.670685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.670660 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "kube-api-access-pnptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.670871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.670730 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token" pod "82ff1268-d948-4eb5-807b-3e4f9c548f1b" (UID: "82ff1268-d948-4eb5-807b-3e4f9c548f1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.671611 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy" pod "f1178048-56ee-4374-a2ed-a9478f96705c" (UID: "f1178048-56ee-4374-a2ed-a9478f96705c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.673236 7074 generic.go:356] "Generic (PLEG): container finished" podID="626a2acd-241b-46ef-a171-481512229bf3" containerID="e2f55d2d1896d21fab87681b86e3a8798e07a9d8d3ca6902a338dc0a04060165" exitCode=0 Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.673329 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" event={"ID":"626a2acd-241b-46ef-a171-481512229bf3","Type":"ContainerDied","Data":"e2f55d2d1896d21fab87681b86e3a8798e07a9d8d3ca6902a338dc0a04060165"} Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.673421 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" event={"ID":"626a2acd-241b-46ef-a171-481512229bf3","Type":"ContainerDied","Data":"d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530"} Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.673577 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2199215ccd4c1fcf41209cb741cf40e5e38d082178a255363156b8d08898530" Oct 31 23:22:07.674426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.673732 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:22:07.676556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.676498 7074 generic.go:356] "Generic (PLEG): container finished" podID="f1178048-56ee-4374-a2ed-a9478f96705c" containerID="763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06" exitCode=0 Oct 31 23:22:07.678083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.677500 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" event={"ID":"f1178048-56ee-4374-a2ed-a9478f96705c","Type":"ContainerDied","Data":"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06"} Oct 31 23:22:07.678083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.677576 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" event={"ID":"f1178048-56ee-4374-a2ed-a9478f96705c","Type":"ContainerDied","Data":"21bc22c129fdc035b90d079ecc1086bf5c2f147877cb08c83f06fb434644a163"} Oct 31 23:22:07.678083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.677612 7074 scope.go:122] "RemoveContainer" containerID="763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06" Oct 31 23:22:07.678083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.677784 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn" Oct 31 23:22:07.682567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.682274 7074 generic.go:356] "Generic (PLEG): container finished" podID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerID="e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547" exitCode=0 Oct 31 23:22:07.682567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.682464 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" event={"ID":"82ff1268-d948-4eb5-807b-3e4f9c548f1b","Type":"ContainerDied","Data":"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547"} Oct 31 23:22:07.682567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.682514 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" event={"ID":"82ff1268-d948-4eb5-807b-3e4f9c548f1b","Type":"ContainerDied","Data":"4872701db1966ce47475700cb85582a78377f4e78a33fc844a370eea39d2e2ae"} Oct 31 23:22:07.682800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.682570 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps" Oct 31 23:22:07.701791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.701775 7074 scope.go:122] "RemoveContainer" containerID="763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06" Oct 31 23:22:07.702260 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:07.702235 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06\": container with ID starting with 763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06 not found: ID does not exist" containerID="763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06" Oct 31 23:22:07.702399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.702375 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06"} err="failed to get container status \"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06\": rpc error: code = NotFound desc = could not find container \"763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06\": container with ID starting with 763b1bf14b764e7856f504ef7dca344506f1c665d8d686ffb1d1173156781a06 not found: ID does not exist" Oct 31 23:22:07.702481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.702472 7074 scope.go:122] "RemoveContainer" containerID="e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547" Oct 31 23:22:07.716708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.716690 7074 scope.go:122] "RemoveContainer" containerID="e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547" Oct 31 23:22:07.717061 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:07.717030 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547\": container with ID starting with e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547 not found: ID does not exist" containerID="e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547" Oct 31 23:22:07.717152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.717063 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547"} err="failed to get container status \"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547\": rpc error: code = NotFound desc = could not find container \"e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547\": container with ID starting with e3aea83fa2fe0a76fbdc2d508de60fd2993413027c4c339541205596e2365547 not found: ID does not exist" Oct 31 23:22:07.756399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756363 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.756569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756455 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.756640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756610 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.756703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756663 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.756703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756659 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.756812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756711 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.756871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756834 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.756972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.756953 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.757069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.757011 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.757069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.757041 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.757175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.757117 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\") pod \"626a2acd-241b-46ef-a171-481512229bf3\" (UID: \"626a2acd-241b-46ef-a171-481512229bf3\") " Oct 31 23:22:07.757175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.757136 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:07.758072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758044 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.758255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758235 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.758435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758396 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.758435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758421 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758435 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758453 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758467 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758481 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758495 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758508 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnptj\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-kube-api-access-pnptj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758521 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758534 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626a2acd-241b-46ef-a171-481512229bf3-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758547 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758562 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758575 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1178048-56ee-4374-a2ed-a9478f96705c-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758588 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758603 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frbfw\" (UniqueName: \"kubernetes.io/projected/82ff1268-d948-4eb5-807b-3e4f9c548f1b-kube-api-access-frbfw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758618 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1178048-56ee-4374-a2ed-a9478f96705c-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758631 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1178048-56ee-4374-a2ed-a9478f96705c-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.763195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.758645 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ff1268-d948-4eb5-807b-3e4f9c548f1b-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.764686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.764651 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:07.765601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.765564 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.765874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.765835 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:07.766161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.766124 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc" pod "626a2acd-241b-46ef-a171-481512229bf3" (UID: "626a2acd-241b-46ef-a171-481512229bf3"). InnerVolumeSpecName "kube-api-access-l2wsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:07.806700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.806659 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:22:07.819029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.818961 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-sd2fn"] Oct 31 23:22:07.859315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.859279 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:22:07.860857 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.860831 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626a2acd-241b-46ef-a171-481512229bf3-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.860935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.860864 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wsc\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-kube-api-access-l2wsc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.860935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.860881 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626a2acd-241b-46ef-a171-481512229bf3-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.860935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.860895 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626a2acd-241b-46ef-a171-481512229bf3-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:07.864230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:07.864201 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-pfzps"] Oct 31 23:22:08.145142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:08.145114 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" path="/var/lib/kubelet/pods/82ff1268-d948-4eb5-807b-3e4f9c548f1b/volumes" Oct 31 23:22:08.145870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:08.145835 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" path="/var/lib/kubelet/pods/f1178048-56ee-4374-a2ed-a9478f96705c/volumes" Oct 31 23:22:08.693648 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:08.693618 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj" Oct 31 23:22:08.746134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:08.746100 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:22:08.760131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:08.760105 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-4j2kj"] Oct 31 23:22:10.144755 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:10.144722 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626a2acd-241b-46ef-a171-481512229bf3" path="/var/lib/kubelet/pods/626a2acd-241b-46ef-a171-481512229bf3/volumes" Oct 31 23:22:12.582255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:12.582216 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:22:12.632157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:12.632126 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:22:14.894934 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:14.894893 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:22:17.583976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:17.583861 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" containerID="cri-o://e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d" gracePeriod=25 Oct 31 23:22:17.633391 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:17.633316 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" containerID="cri-o://2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489" gracePeriod=25 Oct 31 23:22:18.624323 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:18.624284 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:22:18.624777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:18.624331 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:22:22.534057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:22.534024 7074 scope.go:122] "RemoveContainer" containerID="e2f55d2d1896d21fab87681b86e3a8798e07a9d8d3ca6902a338dc0a04060165" Oct 31 23:22:22.548257 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:22.548234 7074 scope.go:122] "RemoveContainer" containerID="a0580f93666ccefdaa29134a8b7f681e5550f34f9c88d05e824ed0f4acb4107a" Oct 31 23:22:26.576946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.576907 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:22:26.577821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577803 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" Oct 31 23:22:26.577871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577822 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" Oct 31 23:22:26.577871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577839 7074 cpu_manager.go:414] "containerMap: removing container" podUID="a6a39603-18f1-4383-b261-bbf664a67e61" containerName="maas-api" Oct 31 23:22:26.577871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577844 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a39603-18f1-4383-b261-bbf664a67e61" containerName="maas-api" Oct 31 23:22:26.577871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577857 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" Oct 31 23:22:26.577871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577863 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577875 7074 cpu_manager.go:414] "containerMap: removing container" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577880 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577904 7074 cpu_manager.go:414] "containerMap: removing container" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577910 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577924 7074 cpu_manager.go:414] "containerMap: removing container" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577928 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577940 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" containerName="maas-api" Oct 31 23:22:26.578095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.577944 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" containerName="maas-api" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578123 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cf81c8-dfcb-4c44-92fa-a9033644e2c3" containerName="payload-processing" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578135 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a39603-18f1-4383-b261-bbf664a67e61" containerName="maas-api" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578144 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1178048-56ee-4374-a2ed-a9478f96705c" containerName="istio-proxy" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578152 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="626a2acd-241b-46ef-a171-481512229bf3" containerName="istio-proxy" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578162 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="514f3f01-5825-420b-93f1-8e8a294ccb70" containerName="payload-pre-processing" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578168 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ff1268-d948-4eb5-807b-3e4f9c548f1b" containerName="istio-proxy" Oct 31 23:22:26.578378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.578175 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb3d8a-3d57-46a7-9924-ce3bbbd536a1" containerName="maas-api" Oct 31 23:22:26.584486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.584464 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.624972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.624933 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:22:26.665458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.665423 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-270e8c-openshift-default-dockercfg-sbdqn\"" Oct 31 23:22:26.681396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681360 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9lz\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.681576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681556 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.681627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681606 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.681782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681759 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.681954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681925 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.682114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.681970 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.682114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.682073 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.682246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.682136 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.682246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.682196 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.783705 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783653 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.783902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783781 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.783902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783809 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.783902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783834 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.783902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783870 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783909 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.783971 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9lz\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784177 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784119 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784176 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784497 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784567 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784592 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784734 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784650 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.784838 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.784820 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.786296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.786271 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.786610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.786581 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.801587 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.801552 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.801968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.801944 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9lz\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.839924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.839849 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:22:26.845934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.845915 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.858366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.858340 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-885fdf-openshift-default-dockercfg-9mwfb\"" Oct 31 23:22:26.866146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.866118 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:22:26.885899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.885825 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.886124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.885938 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhnj\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.886124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.885984 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.886265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.886187 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.886417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.886392 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.887474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.886768 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.887474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.886866 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.887474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.887060 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.887474 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.887120 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.900905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.900872 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:26.990003 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.989710 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.990003 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.989797 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994308 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994458 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994532 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994567 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994662 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhnj\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.994716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994717 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.996097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.994791 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.996097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.995457 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.996097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.995727 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:26.996097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.995965 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.000041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.996354 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.000041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.998229 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.000041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:26.999047 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.005733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.005697 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.014061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.012445 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhnj\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.024579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.024329 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-qswks\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.169666 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.169576 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:27.212833 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:22:27.212322 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8285b23_0cb3_409c_8bcd_e933d1e359e1.slice/crio-84f78492d845b9c3c8cb10932ce39d51937ef00477ad6a8fa9b19778a0244f44 WatchSource:0}: Error finding container 84f78492d845b9c3c8cb10932ce39d51937ef00477ad6a8fa9b19778a0244f44: Status 404 returned error can't find the container with id 84f78492d845b9c3c8cb10932ce39d51937ef00477ad6a8fa9b19778a0244f44 Oct 31 23:22:27.212833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.212345 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:22:27.215829 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.215784 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.215949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.215874 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.215949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.215912 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.496153 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:22:27.496083 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105e3e1e_8500_4de6_b605_a5f79adb683a.slice/crio-56b43083548536e9c3b2acc4c05e580502a7c906a6cb57fde10b69affc0baf73 WatchSource:0}: Error finding container 56b43083548536e9c3b2acc4c05e580502a7c906a6cb57fde10b69affc0baf73: Status 404 returned error can't find the container with id 56b43083548536e9c3b2acc4c05e580502a7c906a6cb57fde10b69affc0baf73 Oct 31 23:22:27.497687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.497615 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:22:27.500215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.500166 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.500383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.500349 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.500482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.500457 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:27.903680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.903622 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" event={"ID":"f8285b23-0cb3-409c-8bcd-e933d1e359e1","Type":"ContainerStarted","Data":"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1"} Oct 31 23:22:27.904230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.903742 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" event={"ID":"f8285b23-0cb3-409c-8bcd-e933d1e359e1","Type":"ContainerStarted","Data":"84f78492d845b9c3c8cb10932ce39d51937ef00477ad6a8fa9b19778a0244f44"} Oct 31 23:22:27.906101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.905982 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" event={"ID":"105e3e1e-8500-4de6-b605-a5f79adb683a","Type":"ContainerStarted","Data":"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a"} Oct 31 23:22:27.906248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.906141 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" event={"ID":"105e3e1e-8500-4de6-b605-a5f79adb683a","Type":"ContainerStarted","Data":"56b43083548536e9c3b2acc4c05e580502a7c906a6cb57fde10b69affc0baf73"} Oct 31 23:22:27.937110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:27.937045 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podStartSLOduration=1.937026833 podStartE2EDuration="1.937026833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:27.934611391 +0000 UTC m=+1925.917353770" watchObservedRunningTime="2026-10-31 23:22:27.937026833 +0000 UTC m=+1925.919769214" Oct 31 23:22:28.001669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.001070 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podStartSLOduration=2.001046351 podStartE2EDuration="2.001046351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:27.964639481 +0000 UTC m=+1925.947381862" watchObservedRunningTime="2026-10-31 23:22:28.001046351 +0000 UTC m=+1925.983788731" Oct 31 23:22:28.132141 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.130177 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:22:28.132141 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.130436 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" podUID="ab793972-6ecb-4f23-bf0c-15407489fa80" containerName="maas-api" containerID="cri-o://a6c3f8a9dd232fc2c5d1c8b1174914f5e4a9cabdd6bde71acac11634e2d4a4c0" gracePeriod=30 Oct 31 23:22:28.200211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.200175 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:28.200485 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.200457 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:28.222850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.221262 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:28.729670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.728407 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:22:28.817019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.791049 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:22:28.944043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.934106 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:28.944043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.934253 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:28.944043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:28.934301 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:29.210139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.209582 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:29.210139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.209659 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:29.225228 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.225190 7074 generic.go:356] "Generic (PLEG): container finished" podID="ab793972-6ecb-4f23-bf0c-15407489fa80" containerID="a6c3f8a9dd232fc2c5d1c8b1174914f5e4a9cabdd6bde71acac11634e2d4a4c0" exitCode=0 Oct 31 23:22:29.226358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.225315 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" event={"ID":"ab793972-6ecb-4f23-bf0c-15407489fa80","Type":"ContainerDied","Data":"a6c3f8a9dd232fc2c5d1c8b1174914f5e4a9cabdd6bde71acac11634e2d4a4c0"} Oct 31 23:22:29.410037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.410008 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:22:29.442941 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.442903 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\") pod \"ab793972-6ecb-4f23-bf0c-15407489fa80\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " Oct 31 23:22:29.467811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.467720 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\") pod \"ab793972-6ecb-4f23-bf0c-15407489fa80\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " Oct 31 23:22:29.481191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.481152 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\") pod \"ab793972-6ecb-4f23-bf0c-15407489fa80\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " Oct 31 23:22:29.505445 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.505415 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\" (UniqueName: \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\") pod \"ab793972-6ecb-4f23-bf0c-15407489fa80\" (UID: \"ab793972-6ecb-4f23-bf0c-15407489fa80\") " Oct 31 23:22:29.505586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.505553 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls" pod "ab793972-6ecb-4f23-bf0c-15407489fa80" (UID: "ab793972-6ecb-4f23-bf0c-15407489fa80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:29.506610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.506586 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:29.510020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.508209 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca" pod "ab793972-6ecb-4f23-bf0c-15407489fa80" (UID: "ab793972-6ecb-4f23-bf0c-15407489fa80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:29.510278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.510256 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r" pod "ab793972-6ecb-4f23-bf0c-15407489fa80" (UID: "ab793972-6ecb-4f23-bf0c-15407489fa80"). InnerVolumeSpecName "kube-api-access-qbj9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:29.511250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.511222 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs" pod "ab793972-6ecb-4f23-bf0c-15407489fa80" (UID: "ab793972-6ecb-4f23-bf0c-15407489fa80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:29.616798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.613525 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab793972-6ecb-4f23-bf0c-15407489fa80-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:29.616798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.616709 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab793972-6ecb-4f23-bf0c-15407489fa80-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:29.616798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.616741 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbj9r\" (UniqueName: \"kubernetes.io/projected/ab793972-6ecb-4f23-bf0c-15407489fa80-kube-api-access-qbj9r\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:29.902382 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.902343 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:29.902558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:29.902406 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:30.207215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.207116 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.207215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.207174 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:30.236974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.236945 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" Oct 31 23:22:30.237302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.237279 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258" event={"ID":"ab793972-6ecb-4f23-bf0c-15407489fa80","Type":"ContainerDied","Data":"7e77308432a2063c668e7e9aa31a28bb2065a63ad5c8003a3bad7a8cb1a93adf"} Oct 31 23:22:30.237407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.237394 7074 scope.go:122] "RemoveContainer" containerID="a6c3f8a9dd232fc2c5d1c8b1174914f5e4a9cabdd6bde71acac11634e2d4a4c0" Oct 31 23:22:30.301758 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:30.301715 7074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab793972_6ecb_4f23_bf0c_15407489fa80.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:30.396196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.396106 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:22:30.409409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.405560 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-36a15fc5-995dbb8f8-lh258"] Oct 31 23:22:30.901821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.901662 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.901821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:30.901735 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:31.179076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:31.178146 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:31.179076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:31.178214 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:31.902068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:31.902039 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:31.902546 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:31.902521 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:32.172646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:32.172550 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.173019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:32.172939 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:32.356615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:32.356571 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab793972-6ecb-4f23-bf0c-15407489fa80" path="/var/lib/kubelet/pods/ab793972-6ecb-4f23-bf0c-15407489fa80/volumes" Oct 31 23:22:32.910037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:32.909955 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.910558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:32.910035 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:33.240120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:33.240034 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:33.240120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:33.240084 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:33.901328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:33.901295 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 23:22:33.901551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:33.901530 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 23:22:34.179495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:34.179396 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-qswks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 23:22:34.190709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:34.190656 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 23:22:34.907341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:34.907303 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:34.907856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:34.907828 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:34.908526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:34.908509 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:22:35.175350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:35.175269 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:35.175671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:35.175650 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:35.176314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:35.176295 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:22:38.041762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.041723 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:22:38.125869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.125843 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:22:38.185434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185332 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\") pod \"a30089af-389e-4d7f-8681-9fde7bfeba30\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " Oct 31 23:22:38.185434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185403 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\") pod \"148d362e-8159-43a4-890a-c049be03ac0e\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " Oct 31 23:22:38.185434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185427 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\" (UniqueName: \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\") pod \"148d362e-8159-43a4-890a-c049be03ac0e\" (UID: \"148d362e-8159-43a4-890a-c049be03ac0e\") " Oct 31 23:22:38.185734 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185620 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\" (UniqueName: \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\") pod \"a30089af-389e-4d7f-8681-9fde7bfeba30\" (UID: \"a30089af-389e-4d7f-8681-9fde7bfeba30\") " Oct 31 23:22:38.185804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185779 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume" pod "a30089af-389e-4d7f-8681-9fde7bfeba30" (UID: "a30089af-389e-4d7f-8681-9fde7bfeba30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:38.185856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.185840 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume" pod "148d362e-8159-43a4-890a-c049be03ac0e" (UID: "148d362e-8159-43a4-890a-c049be03ac0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:38.186558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.186530 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30089af-389e-4d7f-8681-9fde7bfeba30-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:38.186686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.186563 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148d362e-8159-43a4-890a-c049be03ac0e-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:38.187884 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.187861 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv" pod "148d362e-8159-43a4-890a-c049be03ac0e" (UID: "148d362e-8159-43a4-890a-c049be03ac0e"). InnerVolumeSpecName "kube-api-access-vpkcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:38.188272 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.188252 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m" pod "a30089af-389e-4d7f-8681-9fde7bfeba30" (UID: "a30089af-389e-4d7f-8681-9fde7bfeba30"). InnerVolumeSpecName "kube-api-access-vgc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:38.288645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.288602 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkcv\" (UniqueName: \"kubernetes.io/projected/148d362e-8159-43a4-890a-c049be03ac0e-kube-api-access-vpkcv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:38.288645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.288653 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgc4m\" (UniqueName: \"kubernetes.io/projected/a30089af-389e-4d7f-8681-9fde7bfeba30-kube-api-access-vgc4m\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:38.476918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.476818 7074 generic.go:356] "Generic (PLEG): container finished" podID="148d362e-8159-43a4-890a-c049be03ac0e" containerID="2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489" exitCode=0 Oct 31 23:22:38.476918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.476887 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" Oct 31 23:22:38.477258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.476983 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" event={"ID":"148d362e-8159-43a4-890a-c049be03ac0e","Type":"ContainerDied","Data":"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489"} Oct 31 23:22:38.477258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.477097 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6" event={"ID":"148d362e-8159-43a4-890a-c049be03ac0e","Type":"ContainerDied","Data":"1518b993170114070b0d1fa61052b4f26f0ab0078c33d51cc0865d1297733296"} Oct 31 23:22:38.477258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.477131 7074 scope.go:122] "RemoveContainer" containerID="2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489" Oct 31 23:22:38.481075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.480697 7074 generic.go:356] "Generic (PLEG): container finished" podID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerID="e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d" exitCode=0 Oct 31 23:22:38.481075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.480816 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" event={"ID":"a30089af-389e-4d7f-8681-9fde7bfeba30","Type":"ContainerDied","Data":"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d"} Oct 31 23:22:38.481075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.480866 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" event={"ID":"a30089af-389e-4d7f-8681-9fde7bfeba30","Type":"ContainerDied","Data":"bae055892a3f8b29d02db7af6ac2b2956bc0595d7bab60ee3388ad737ac7c7a7"} Oct 31 23:22:38.481075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.480948 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj" Oct 31 23:22:38.504136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.494966 7074 scope.go:122] "RemoveContainer" containerID="2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489" Oct 31 23:22:38.504136 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:38.495650 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489\": container with ID starting with 2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489 not found: ID does not exist" containerID="2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489" Oct 31 23:22:38.504136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.495689 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489"} err="failed to get container status \"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489\": rpc error: code = NotFound desc = could not find container \"2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489\": container with ID starting with 2b6d22c61d43da0255b6c0d6ed51319c7b319ba9f4745718cf018c7706f63489 not found: ID does not exist" Oct 31 23:22:38.504136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.495714 7074 scope.go:122] "RemoveContainer" containerID="e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d" Oct 31 23:22:38.519491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.519447 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:22:38.520441 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.520418 7074 scope.go:122] "RemoveContainer" containerID="e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d" Oct 31 23:22:38.520844 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:38.520817 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d\": container with ID starting with e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d not found: ID does not exist" containerID="e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d" Oct 31 23:22:38.521004 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.520966 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d"} err="failed to get container status \"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d\": rpc error: code = NotFound desc = could not find container \"e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d\": container with ID starting with e566ccc45f265d33c4690d966b3907776c77819235c69872d2c3f6c860ed1b9d not found: ID does not exist" Oct 31 23:22:38.526195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.526164 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a16dfb-c4c48959f-w9vw6"] Oct 31 23:22:38.549946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.549910 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:22:38.557836 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:38.557802 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a16dfb-5dd4d8855d-mmgsj"] Oct 31 23:22:40.146618 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:40.146579 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148d362e-8159-43a4-890a-c049be03ac0e" path="/var/lib/kubelet/pods/148d362e-8159-43a4-890a-c049be03ac0e/volumes" Oct 31 23:22:40.147139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:40.147123 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" path="/var/lib/kubelet/pods/a30089af-389e-4d7f-8681-9fde7bfeba30/volumes" Oct 31 23:22:41.567039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:41.566981 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:22:41.567874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:41.567836 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" podUID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" containerName="maas-api" containerID="cri-o://edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15" gracePeriod=30 Oct 31 23:22:42.188640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.187242 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.221498 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\") pod \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.221557 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\") pod \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.222467 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\") pod \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.224718 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca" pod "69ef6af7-eb72-45dc-9106-ef21889ab2f2" (UID: "69ef6af7-eb72-45dc-9106-ef21889ab2f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.225218 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\" (UniqueName: \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\") pod \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\" (UID: \"69ef6af7-eb72-45dc-9106-ef21889ab2f2\") " Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.229380 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef6af7-eb72-45dc-9106-ef21889ab2f2-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:42.230037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.229842 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls" pod "69ef6af7-eb72-45dc-9106-ef21889ab2f2" (UID: "69ef6af7-eb72-45dc-9106-ef21889ab2f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:42.250044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.243231 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs" pod "69ef6af7-eb72-45dc-9106-ef21889ab2f2" (UID: "69ef6af7-eb72-45dc-9106-ef21889ab2f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:42.250044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.244161 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq" pod "69ef6af7-eb72-45dc-9106-ef21889ab2f2" (UID: "69ef6af7-eb72-45dc-9106-ef21889ab2f2"). InnerVolumeSpecName "kube-api-access-hxwmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:42.336694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.336648 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwmq\" (UniqueName: \"kubernetes.io/projected/69ef6af7-eb72-45dc-9106-ef21889ab2f2-kube-api-access-hxwmq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:42.336694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.336693 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:42.336694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.336709 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ef6af7-eb72-45dc-9106-ef21889ab2f2-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:42.523601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.523558 7074 generic.go:356] "Generic (PLEG): container finished" podID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" containerID="edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15" exitCode=0 Oct 31 23:22:42.523601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.523603 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" Oct 31 23:22:42.523850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.523641 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" event={"ID":"69ef6af7-eb72-45dc-9106-ef21889ab2f2","Type":"ContainerDied","Data":"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15"} Oct 31 23:22:42.523850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.523696 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4" event={"ID":"69ef6af7-eb72-45dc-9106-ef21889ab2f2","Type":"ContainerDied","Data":"c8f8e20468dbbf11d663c2d8c3d4ab6a3e898757db997932e3095d777598016e"} Oct 31 23:22:42.523850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.523720 7074 scope.go:122] "RemoveContainer" containerID="edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15" Oct 31 23:22:42.537097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.536944 7074 scope.go:122] "RemoveContainer" containerID="edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15" Oct 31 23:22:42.537442 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:42.537374 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15\": container with ID starting with edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15 not found: ID does not exist" containerID="edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15" Oct 31 23:22:42.537442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.537418 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15"} err="failed to get container status \"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15\": rpc error: code = NotFound desc = could not find container \"edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15\": container with ID starting with edd787e7162f601265921fc9aab1931babeeebe299d4565cd12d672054726f15 not found: ID does not exist" Oct 31 23:22:42.582525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.582487 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:22:42.592405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.592371 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a16dfb-699b99d5bf-mvpn4"] Oct 31 23:22:42.894535 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.894430 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:22:42.895802 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895778 7074 cpu_manager.go:414] "containerMap: removing container" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" Oct 31 23:22:42.895802 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895799 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895816 7074 cpu_manager.go:414] "containerMap: removing container" podUID="ab793972-6ecb-4f23-bf0c-15407489fa80" containerName="maas-api" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895824 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab793972-6ecb-4f23-bf0c-15407489fa80" containerName="maas-api" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895853 7074 cpu_manager.go:414] "containerMap: removing container" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895861 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895923 7074 cpu_manager.go:414] "containerMap: removing container" podUID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" containerName="maas-api" Oct 31 23:22:42.896073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.895930 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" containerName="maas-api" Oct 31 23:22:42.896473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.896132 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" containerName="maas-api" Oct 31 23:22:42.896473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.896152 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30089af-389e-4d7f-8681-9fde7bfeba30" containerName="payload-processing" Oct 31 23:22:42.896473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.896164 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab793972-6ecb-4f23-bf0c-15407489fa80" containerName="maas-api" Oct 31 23:22:42.896473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.896177 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="148d362e-8159-43a4-890a-c049be03ac0e" containerName="payload-pre-processing" Oct 31 23:22:42.902006 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.901971 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.911671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.911636 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:22:42.915569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.915534 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-479107-openshift-default-dockercfg-pnn5x\"" Oct 31 23:22:42.948155 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948116 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948243 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948368 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948527 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948610 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948669 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtkf\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948760 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948891 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:42.948945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:42.948928 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051054 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051145 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051233 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051329 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051388 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051439 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtkf\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051500 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051507 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051534 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051572 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.051703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051665 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.052143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.051711 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.052143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.052027 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.052547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.052527 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.054370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.054342 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.054445 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.054378 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.066494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.066464 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.066785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.066764 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtkf\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.229888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.229785 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:43.789552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.789512 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:22:43.793745 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:22:43.793713 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fa2cb1_9ca3_4868_9409_3da5abce30de.slice/crio-d4b5e3ce5583e1f7e552b42ee323f1b6f4f9ec01372b601c56e52635089906ea WatchSource:0}: Error finding container d4b5e3ce5583e1f7e552b42ee323f1b6f4f9ec01372b601c56e52635089906ea: Status 404 returned error can't find the container with id d4b5e3ce5583e1f7e552b42ee323f1b6f4f9ec01372b601c56e52635089906ea Oct 31 23:22:43.854060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.851406 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:43.854060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.851527 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:43.854060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:43.851579 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:22:44.151786 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:44.151678 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ef6af7-eb72-45dc-9106-ef21889ab2f2" path="/var/lib/kubelet/pods/69ef6af7-eb72-45dc-9106-ef21889ab2f2/volumes" Oct 31 23:22:44.574327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:44.574290 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" event={"ID":"24fa2cb1-9ca3-4868-9409-3da5abce30de","Type":"ContainerStarted","Data":"0f096ea9aaf6fd47941c00f785766b785d9de58d8a2acf40abcff275eaf32699"} Oct 31 23:22:44.574486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:44.574355 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" event={"ID":"24fa2cb1-9ca3-4868-9409-3da5abce30de","Type":"ContainerStarted","Data":"d4b5e3ce5583e1f7e552b42ee323f1b6f4f9ec01372b601c56e52635089906ea"} Oct 31 23:22:44.610173 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:44.610116 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podStartSLOduration=2.610099114 podStartE2EDuration="2.610099114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:44.609558696 +0000 UTC m=+1942.592301075" watchObservedRunningTime="2026-10-31 23:22:44.610099114 +0000 UTC m=+1942.592841491" Oct 31 23:22:44.916868 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:44.916763 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:22:45.230356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:45.230270 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:45.231824 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:45.231797 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:45.231953 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:45.231883 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:46.230740 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:46.230703 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:46.231246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:46.230756 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:47.246075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:47.243945 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:47.246075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:47.244038 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:48.231167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.231133 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:48.231369 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.231199 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:48.394307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.394271 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:22:48.395121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.395088 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" containerID="cri-o://979c62b871b897cb3eadd0b1d8e1593c994a1acfad68490ddda48bd02fe59eb1" gracePeriod=30 Oct 31 23:22:48.415538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.399200 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:22:48.415538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.399464 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" containerID="cri-o://7c257471b7060a622c45eac4961dd4b138fe82057feed974731df1aa08bd20c1" gracePeriod=30 Oct 31 23:22:48.415538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.412896 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:22:48.415538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:48.413187 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" containerID="cri-o://54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf" gracePeriod=30 Oct 31 23:22:49.230528 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:49.230487 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:49.230710 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:49.230546 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:50.231175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:50.231136 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 23:22:50.231596 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:50.231205 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 23:22:51.267011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:51.266939 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:51.267495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:51.267435 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:51.268532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:51.268510 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:22:52.745464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:52.745429 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:22:52.745900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:52.745488 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:22:53.088963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.088916 7074 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:22:53.089173 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.089038 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:22:53.815248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.815215 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:22:53.860112 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.860076 7074 generic.go:356] "Generic (PLEG): container finished" podID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerID="7c257471b7060a622c45eac4961dd4b138fe82057feed974731df1aa08bd20c1" exitCode=0 Oct 31 23:22:53.860289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.860147 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" event={"ID":"3cd97fdf-42aa-4d51-a2a3-65e362339cab","Type":"ContainerDied","Data":"7c257471b7060a622c45eac4961dd4b138fe82057feed974731df1aa08bd20c1"} Oct 31 23:22:53.860289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.860233 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" event={"ID":"3cd97fdf-42aa-4d51-a2a3-65e362339cab","Type":"ContainerDied","Data":"346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8"} Oct 31 23:22:53.860289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.860284 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346ac2fd4f8919288d18d1a1aacb6cc0ee713f28adfa464d7d1c2ca98231edc8" Oct 31 23:22:53.862530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.862502 7074 generic.go:356] "Generic (PLEG): container finished" podID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerID="979c62b871b897cb3eadd0b1d8e1593c994a1acfad68490ddda48bd02fe59eb1" exitCode=0 Oct 31 23:22:53.863363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.863335 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" event={"ID":"d6650d91-acb8-4428-93e2-fbeea25cbd13","Type":"ContainerDied","Data":"979c62b871b897cb3eadd0b1d8e1593c994a1acfad68490ddda48bd02fe59eb1"} Oct 31 23:22:53.863686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.863665 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:22:53.865180 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.865154 7074 generic.go:356] "Generic (PLEG): container finished" podID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerID="54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf" exitCode=0 Oct 31 23:22:53.865292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.865191 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" Oct 31 23:22:53.865292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.865218 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" event={"ID":"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9","Type":"ContainerDied","Data":"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf"} Oct 31 23:22:53.865292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.865260 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb" event={"ID":"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9","Type":"ContainerDied","Data":"a1294225b29a79ce11f7faf879b66ca73beb88b353d284c9896c728942b2914c"} Oct 31 23:22:53.865292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.865289 7074 scope.go:122] "RemoveContainer" containerID="54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf" Oct 31 23:22:53.877168 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.877145 7074 scope.go:122] "RemoveContainer" containerID="54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf" Oct 31 23:22:53.877511 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:53.877490 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf\": container with ID starting with 54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf not found: ID does not exist" containerID="54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf" Oct 31 23:22:53.877583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.877520 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf"} err="failed to get container status \"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf\": rpc error: code = NotFound desc = could not find container \"54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf\": container with ID starting with 54478045557b57b8c6fb4314f90f3d38d39d488d76f45ec4354676c77439c4cf not found: ID does not exist" Oct 31 23:22:53.907598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907550 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.907797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907673 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.907797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907742 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.907929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907802 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.907929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907835 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.908064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.907943 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.908064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.908026 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.908187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.908061 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.908187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.908174 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\") pod \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\" (UID: \"1bb1d84f-ae46-43b0-b356-1a6dc2f815f9\") " Oct 31 23:22:53.910515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.908678 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:53.910515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.909341 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:53.911110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.910767 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:53.911661 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.911640 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:53.911983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.911964 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:53.912272 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.912246 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:53.912488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.912467 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:53.913138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.913106 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "kube-api-access-629ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:53.916209 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.916179 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:53.916555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:53.916533 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo" pod "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" (UID: "1bb1d84f-ae46-43b0-b356-1a6dc2f815f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:54.010798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.010761 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011053 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011188 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011229 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011284 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011318 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011367 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011473 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.011645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.011515 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\") pod \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\" (UID: \"3cd97fdf-42aa-4d51-a2a3-65e362339cab\") " Oct 31 23:22:54.012082 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.012054 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.012874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.012304 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.012874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.012666 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013261 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013387 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013506 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013530 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013546 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013559 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013573 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-629ql\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-kube-api-access-629ql\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013586 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013594 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013602 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013610 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013618 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013626 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013634 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.014954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.013645 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.016292 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.016272 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.017202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.017172 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "kube-api-access-2nkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:54.017380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.017359 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:54.027118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.027081 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token" pod "3cd97fdf-42aa-4d51-a2a3-65e362339cab" (UID: "3cd97fdf-42aa-4d51-a2a3-65e362339cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:54.109436 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.109408 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:22:54.115252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.115220 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.115252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.115251 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.115252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.115262 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd97fdf-42aa-4d51-a2a3-65e362339cab-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.115515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.115272 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkkp\" (UniqueName: \"kubernetes.io/projected/3cd97fdf-42aa-4d51-a2a3-65e362339cab-kube-api-access-2nkkp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.192629 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.192591 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:22:54.200306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.200269 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-ld8pb"] Oct 31 23:22:54.216376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216341 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.216558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216392 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.216558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216415 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.216558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216448 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.216558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216488 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.216558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216556 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.217471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.216709 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.217471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.217289 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.217471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.217434 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.217693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.217589 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\") pod \"d6650d91-acb8-4428-93e2-fbeea25cbd13\" (UID: \"d6650d91-acb8-4428-93e2-fbeea25cbd13\") " Oct 31 23:22:54.218100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218008 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.218211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.217308 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.218351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218331 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.218400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218353 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:54.218561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218545 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6650d91-acb8-4428-93e2-fbeea25cbd13-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.218623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218563 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.218623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218574 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.218623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218582 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.218623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.218596 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.219644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.219612 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:54.220066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.220042 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "kube-api-access-jzm7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:54.220251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.220232 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:54.220419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.220405 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token" pod "d6650d91-acb8-4428-93e2-fbeea25cbd13" (UID: "d6650d91-acb8-4428-93e2-fbeea25cbd13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:54.321089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.320889 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.321089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.320929 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.321089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.320944 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzm7v\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-kube-api-access-jzm7v\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.321089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.320960 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6650d91-acb8-4428-93e2-fbeea25cbd13-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:54.876611 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.876579 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" Oct 31 23:22:54.877114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.876601 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw" event={"ID":"d6650d91-acb8-4428-93e2-fbeea25cbd13","Type":"ContainerDied","Data":"9c9bbfb584192efd70592f9371873e56d0334268eb3726b63d1838f89c501a35"} Oct 31 23:22:54.877114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.876672 7074 scope.go:122] "RemoveContainer" containerID="979c62b871b897cb3eadd0b1d8e1593c994a1acfad68490ddda48bd02fe59eb1" Oct 31 23:22:54.878274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.878249 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4" Oct 31 23:22:54.903690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.903654 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:22:54.907797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.907768 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-nfqq4"] Oct 31 23:22:54.922696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.922662 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:22:54.926770 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:54.926746 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-wvbgw"] Oct 31 23:22:56.146188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.146154 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" path="/var/lib/kubelet/pods/1bb1d84f-ae46-43b0-b356-1a6dc2f815f9/volumes" Oct 31 23:22:56.146779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.146763 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" path="/var/lib/kubelet/pods/3cd97fdf-42aa-4d51-a2a3-65e362339cab/volumes" Oct 31 23:22:56.147311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.147291 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" path="/var/lib/kubelet/pods/d6650d91-acb8-4428-93e2-fbeea25cbd13/volumes" Oct 31 23:22:56.214543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.214509 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:22:56.214777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.214737 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" podUID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" containerName="maas-api" containerID="cri-o://8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a" gracePeriod=30 Oct 31 23:22:56.330908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.330866 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:22:56.331167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.331135 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" podUID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" containerName="maas-api" containerID="cri-o://6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405" gracePeriod=30 Oct 31 23:22:56.590949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.586535 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:22:56.667192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.667163 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:22:56.703034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.702929 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\") pod \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " Oct 31 23:22:56.703034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.703011 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") pod \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " Oct 31 23:22:56.703282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.703111 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\") pod \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " Oct 31 23:22:56.703282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.703142 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\" (UniqueName: \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\") pod \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\" (UID: \"d917e2f7-2c99-4abe-bba1-083cb4c69e4f\") " Oct 31 23:22:56.703548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.703511 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca" pod "d917e2f7-2c99-4abe-bba1-083cb4c69e4f" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:56.704043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.704017 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.705904 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.705877 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls" pod "d917e2f7-2c99-4abe-bba1-083cb4c69e4f" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:56.705983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.705944 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c" pod "d917e2f7-2c99-4abe-bba1-083cb4c69e4f" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f"). InnerVolumeSpecName "kube-api-access-62m4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:56.706280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.706257 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs" pod "d917e2f7-2c99-4abe-bba1-083cb4c69e4f" (UID: "d917e2f7-2c99-4abe-bba1-083cb4c69e4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.805175 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\") pod \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.805279 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\" (UniqueName: \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\") pod \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.805361 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") pod \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.805671 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\") pod \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\" (UID: \"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88\") " Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.806472 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.806498 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.806813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.806513 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62m4c\" (UniqueName: \"kubernetes.io/projected/d917e2f7-2c99-4abe-bba1-083cb4c69e4f-kube-api-access-62m4c\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.808897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.808870 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca" pod "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:56.811547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.811501 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf" pod "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88"). InnerVolumeSpecName "kube-api-access-w9gvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:56.832024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.831037 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs" pod "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:56.858273 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.858219 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls" pod "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" (UID: "438d0ae1-b63e-47a7-bfa3-39e9c66c3a88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:56.973277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.973060 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.973277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.973112 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.973277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.973129 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gvf\" (UniqueName: \"kubernetes.io/projected/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-kube-api-access-w9gvf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.973277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.973146 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.991015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.973843 7074 generic.go:356] "Generic (PLEG): container finished" podID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" containerID="8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a" exitCode=0 Oct 31 23:22:56.991015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.974121 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" event={"ID":"d917e2f7-2c99-4abe-bba1-083cb4c69e4f","Type":"ContainerDied","Data":"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a"} Oct 31 23:22:56.991015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.974180 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" event={"ID":"d917e2f7-2c99-4abe-bba1-083cb4c69e4f","Type":"ContainerDied","Data":"0091cdbe526bb1e90c380f3de6691e69460e27bc63f966b069968c69760796ec"} Oct 31 23:22:56.991015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.974209 7074 scope.go:122] "RemoveContainer" containerID="8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a" Oct 31 23:22:56.991015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:56.974334 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq" Oct 31 23:22:57.045012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.028863 7074 generic.go:356] "Generic (PLEG): container finished" podID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" containerID="6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405" exitCode=0 Oct 31 23:22:57.045012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.029054 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" event={"ID":"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88","Type":"ContainerDied","Data":"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405"} Oct 31 23:22:57.045012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.029110 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" event={"ID":"438d0ae1-b63e-47a7-bfa3-39e9c66c3a88","Type":"ContainerDied","Data":"3c1698bc5d17a936224a1fe02406dc548445adfd769970d0e4eb05a9db8a061e"} Oct 31 23:22:57.045012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.029194 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj" Oct 31 23:22:57.108060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.108028 7074 scope.go:122] "RemoveContainer" containerID="8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a" Oct 31 23:22:57.109841 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:57.109801 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a\": container with ID starting with 8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a not found: ID does not exist" containerID="8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a" Oct 31 23:22:57.110039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.109849 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a"} err="failed to get container status \"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a\": rpc error: code = NotFound desc = could not find container \"8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a\": container with ID starting with 8d7ec25872ed9724e3a84e9a91a2732efc42fb81f7e13119d4fb1a2259514f0a not found: ID does not exist" Oct 31 23:22:57.110039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.109873 7074 scope.go:122] "RemoveContainer" containerID="6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405" Oct 31 23:22:57.125107 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.125077 7074 scope.go:122] "RemoveContainer" containerID="6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405" Oct 31 23:22:57.125476 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:22:57.125452 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405\": container with ID starting with 6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405 not found: ID does not exist" containerID="6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405" Oct 31 23:22:57.125571 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.125484 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405"} err="failed to get container status \"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405\": rpc error: code = NotFound desc = could not find container \"6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405\": container with ID starting with 6b05ff7e54a837800eefccfc80146914e13394717f984cc1f99edbc50dc32405 not found: ID does not exist" Oct 31 23:22:57.171846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.171809 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:22:57.247097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.247072 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4528ed-556d8b5cf7-6gltj"] Oct 31 23:22:57.255486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.255427 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:22:57.264338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:57.264108 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479107-555b99b64c-ww2mq"] Oct 31 23:22:58.150133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:58.150095 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" path="/var/lib/kubelet/pods/438d0ae1-b63e-47a7-bfa3-39e9c66c3a88/volumes" Oct 31 23:22:58.150884 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:22:58.150864 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" path="/var/lib/kubelet/pods/d917e2f7-2c99-4abe-bba1-083cb4c69e4f/volumes" Oct 31 23:23:05.047252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.047212 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:23:05.047886 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.047573 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" containerID="cri-o://0f096ea9aaf6fd47941c00f785766b785d9de58d8a2acf40abcff275eaf32699" gracePeriod=30 Oct 31 23:23:05.091907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.091869 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:23:05.092733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092710 7074 cpu_manager.go:414] "containerMap: removing container" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" Oct 31 23:23:05.092733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092730 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092755 7074 cpu_manager.go:414] "containerMap: removing container" podUID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" containerName="maas-api" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092761 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" containerName="maas-api" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092776 7074 cpu_manager.go:414] "containerMap: removing container" podUID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" containerName="maas-api" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092782 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" containerName="maas-api" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092797 7074 cpu_manager.go:414] "containerMap: removing container" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092801 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092811 7074 cpu_manager.go:414] "containerMap: removing container" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092815 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" Oct 31 23:23:05.092932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092932 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6650d91-acb8-4428-93e2-fbeea25cbd13" containerName="istio-proxy" Oct 31 23:23:05.093589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092940 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb1d84f-ae46-43b0-b356-1a6dc2f815f9" containerName="istio-proxy" Oct 31 23:23:05.093589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092952 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd97fdf-42aa-4d51-a2a3-65e362339cab" containerName="istio-proxy" Oct 31 23:23:05.093589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092961 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d917e2f7-2c99-4abe-bba1-083cb4c69e4f" containerName="maas-api" Oct 31 23:23:05.093589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.092968 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="438d0ae1-b63e-47a7-bfa3-39e9c66c3a88" containerName="maas-api" Oct 31 23:23:05.099186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.099156 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.104066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.103898 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bf279c-serving-cert\"" Oct 31 23:23:05.108365 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.108332 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:23:05.256015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.250528 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.256015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.250833 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.256015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.250887 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mp22\" (UniqueName: \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.256015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.250978 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.359065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.356659 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.359065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.356934 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.359065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.356980 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mp22\" (UniqueName: \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.359065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.357072 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.359065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.358398 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.383034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.369598 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.383034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.379680 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.437011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.427912 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mp22\" (UniqueName: \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\") pod \"maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:05.727816 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:05.727710 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:06.260124 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:23:06.260078 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54681c2_f396_49eb_a8a2_5c2fd00f3cac.slice/crio-f5f29e279d4aacd6922c0b93ba2e68fce68f76f7740a12559b9a4cd52ed44cd2 WatchSource:0}: Error finding container f5f29e279d4aacd6922c0b93ba2e68fce68f76f7740a12559b9a4cd52ed44cd2: Status 404 returned error can't find the container with id f5f29e279d4aacd6922c0b93ba2e68fce68f76f7740a12559b9a4cd52ed44cd2 Oct 31 23:23:06.261603 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.261568 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:23:06.262592 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.262567 7074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:23:06.269279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.269248 7074 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:23:06.269409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.269306 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:23:06.884709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.884676 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:23:06.893606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.890781 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.908849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.908815 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:23:06.927842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.927794 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.927879 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928003 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928069 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928096 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz9n\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928114 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928264 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928170 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928264 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928220 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.928371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.928300 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:06.993097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:06.993004 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1deb6-openshift-default-dockercfg-sx6st\"" Oct 31 23:23:07.029789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.029747 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.029956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.029810 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.029956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.029849 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz9n\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.029956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.029868 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.029956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.029934 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030016 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030147 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030247 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030271 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030509 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030340 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030509 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030342 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030611 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030552 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030611 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030584 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.030728 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.030710 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.032447 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.032426 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.032812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.032793 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.064687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.064654 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz9n\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.067782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.067753 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.187685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.187607 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" event={"ID":"c54681c2-f396-49eb-a8a2-5c2fd00f3cac","Type":"ContainerStarted","Data":"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8"} Oct 31 23:23:07.187685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.187669 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:07.187685 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.187685 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" event={"ID":"c54681c2-f396-49eb-a8a2-5c2fd00f3cac","Type":"ContainerStarted","Data":"f5f29e279d4aacd6922c0b93ba2e68fce68f76f7740a12559b9a4cd52ed44cd2"} Oct 31 23:23:07.213813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.213756 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" podStartSLOduration=1.76488541 podStartE2EDuration="2.213738618s" totalImagesPullingTime="448.853208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:07.211907435 +0000 UTC m=+1965.194649814" watchObservedRunningTime="2026-10-31 23:23:07.213738618 +0000 UTC m=+1965.196480998" Oct 31 23:23:07.214495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.214470 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:07.515243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.515203 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:23:07.525680 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:23:07.525605 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e23523_c883_45b6_bd79_a5062bf91962.slice/crio-1c21df2bef3c336258e857016dbe15bf0eb73c1fcb692e485099dedc86a369ca WatchSource:0}: Error finding container 1c21df2bef3c336258e857016dbe15bf0eb73c1fcb692e485099dedc86a369ca: Status 404 returned error can't find the container with id 1c21df2bef3c336258e857016dbe15bf0eb73c1fcb692e485099dedc86a369ca Oct 31 23:23:07.528820 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.528778 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:07.528917 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.528857 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:07.528917 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:07.528886 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:08.197614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.197572 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" event={"ID":"80e23523-c883-45b6-bd79-a5062bf91962","Type":"ContainerStarted","Data":"bac2ac4e4c0b373bfffbabe9aece95a1e28b3ac86248021128955dd40b2857a5"} Oct 31 23:23:08.197821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.197629 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" event={"ID":"80e23523-c883-45b6-bd79-a5062bf91962","Type":"ContainerStarted","Data":"1c21df2bef3c336258e857016dbe15bf0eb73c1fcb692e485099dedc86a369ca"} Oct 31 23:23:08.215306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.215270 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:08.216286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.216263 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:08.216401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.216317 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:08.225445 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:08.225392 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podStartSLOduration=2.225377529 podStartE2EDuration="2.225377529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:08.222456775 +0000 UTC m=+1966.205199153" watchObservedRunningTime="2026-10-31 23:23:08.225377529 +0000 UTC m=+1966.208119907" Oct 31 23:23:09.215201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:09.215166 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:09.215620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:09.215220 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:10.215270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.215233 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:10.215842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.215296 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:10.216863 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.216838 7074 generic.go:356] "Generic (PLEG): container finished" podID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerID="0f096ea9aaf6fd47941c00f785766b785d9de58d8a2acf40abcff275eaf32699" exitCode=0 Oct 31 23:23:10.216974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.216914 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" event={"ID":"24fa2cb1-9ca3-4868-9409-3da5abce30de","Type":"ContainerDied","Data":"0f096ea9aaf6fd47941c00f785766b785d9de58d8a2acf40abcff275eaf32699"} Oct 31 23:23:10.884399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.884325 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:23:10.919934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.919897 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.919960 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920128 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920173 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920264 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920292 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920290 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.920503 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920401 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920503 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920452 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920503 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920479 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.920630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920524 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.920630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920538 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:10.920743 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920675 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\") pod \"24fa2cb1-9ca3-4868-9409-3da5abce30de\" (UID: \"24fa2cb1-9ca3-4868-9409-3da5abce30de\") " Oct 31 23:23:10.920743 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.920687 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.921472 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.921502 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.921519 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.921534 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fa2cb1-9ca3-4868-9409-3da5abce30de-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.921549 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.922725 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "kube-api-access-6wtkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.922768 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:10.924452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.922862 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:10.927136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:10.927105 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy" pod "24fa2cb1-9ca3-4868-9409-3da5abce30de" (UID: "24fa2cb1-9ca3-4868-9409-3da5abce30de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:11.023739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.023700 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wtkf\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-kube-api-access-6wtkf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.023739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.023738 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.023926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.023747 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.023926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.023759 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fa2cb1-9ca3-4868-9409-3da5abce30de-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.217885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.217786 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:11.217885 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.217850 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:11.229410 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.229378 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" event={"ID":"24fa2cb1-9ca3-4868-9409-3da5abce30de","Type":"ContainerDied","Data":"d4b5e3ce5583e1f7e552b42ee323f1b6f4f9ec01372b601c56e52635089906ea"} Oct 31 23:23:11.229593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.229438 7074 scope.go:122] "RemoveContainer" containerID="0f096ea9aaf6fd47941c00f785766b785d9de58d8a2acf40abcff275eaf32699" Oct 31 23:23:11.229593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.229567 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn" Oct 31 23:23:11.307040 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.306962 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:23:11.318053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.318021 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-kc6jn"] Oct 31 23:23:11.592610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.590944 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:23:11.592610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.592451 7074 cpu_manager.go:414] "containerMap: removing container" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" Oct 31 23:23:11.592610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.592477 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" Oct 31 23:23:11.592918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.592708 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" containerName="istio-proxy" Oct 31 23:23:11.616696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.616646 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:23:11.626814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.626754 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.652015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.643735 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.652015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.643797 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.652015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.643832 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.652015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.643868 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.675014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.670534 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.675014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.670765 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.675014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.671002 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfgl\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.675014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.671044 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.675014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.671089 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775138 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775293 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775378 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775537 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfgl\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775584 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.775716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775618 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.776208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775778 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.776208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775809 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.776208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.775838 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.777239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.776367 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.777239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.776699 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.777239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.776965 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.777239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.777197 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.778746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.777739 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.780650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.780011 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.780650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.780443 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.790024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.789951 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfgl\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.791628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.791601 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:11.992684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:11.991354 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:12.022884 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.018076 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:23:12.027423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.026077 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:23:12.027423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.026175 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111180 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5725h\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111413 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111382 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111423 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111475 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111602 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111572 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111668 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111751 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.111805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111800 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.112024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.111822 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.176399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.176364 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fa2cb1-9ca3-4868-9409-3da5abce30de" path="/var/lib/kubelet/pods/24fa2cb1-9ca3-4868-9409-3da5abce30de/volumes" Oct 31 23:23:12.213409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213366 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.213604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213467 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.213604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213540 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.213716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213692 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.214005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213853 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.214005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.213971 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.215168 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.214057 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.215168 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.214083 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.215168 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.214153 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5725h\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.215368 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.215174 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.215478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.215437 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.216662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.215906 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.216662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.216407 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.217615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.217198 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:12.217615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.217224 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.217615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.217246 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:12.218831 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.218803 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.220250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.220202 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.231614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.231490 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.232955 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.232748 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5725h\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:12.431423 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:23:12.431378 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4f68aa_fee5_43a3_b3b0_c768f0a3e123.slice/crio-60e3dd84c20b69102910ffcbe78acc030a53beee9c5795522c1d6e7cc0322842 WatchSource:0}: Error finding container 60e3dd84c20b69102910ffcbe78acc030a53beee9c5795522c1d6e7cc0322842: Status 404 returned error can't find the container with id 60e3dd84c20b69102910ffcbe78acc030a53beee9c5795522c1d6e7cc0322842 Oct 31 23:23:12.431621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.431598 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:23:12.434402 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.434363 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:12.434496 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.434433 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:12.434496 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.434489 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:12.478320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:12.478281 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:13.010047 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.010010 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:23:13.016186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.016046 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:13.016186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.016145 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:13.016778 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.016197 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:13.207430 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.207400 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:23:13.215411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.215377 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:13.215553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.215432 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:13.297598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.297494 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" event={"ID":"db4f68aa-fee5-43a3-b3b0-c768f0a3e123","Type":"ContainerStarted","Data":"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036"} Oct 31 23:23:13.297598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.297572 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" event={"ID":"db4f68aa-fee5-43a3-b3b0-c768f0a3e123","Type":"ContainerStarted","Data":"60e3dd84c20b69102910ffcbe78acc030a53beee9c5795522c1d6e7cc0322842"} Oct 31 23:23:13.301113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.301075 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" event={"ID":"13751078-df8a-4c5d-a011-4bf0edc29cb9","Type":"ContainerStarted","Data":"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f"} Oct 31 23:23:13.301251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.301148 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" event={"ID":"13751078-df8a-4c5d-a011-4bf0edc29cb9","Type":"ContainerStarted","Data":"409fc5d45f62d2872ed8f09ff0032b097c246cc7f819b14143c30c63459424aa"} Oct 31 23:23:13.328650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.328588 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podStartSLOduration=2.328568008 podStartE2EDuration="2.328568008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:13.322822295 +0000 UTC m=+1971.305564694" watchObservedRunningTime="2026-10-31 23:23:13.328568008 +0000 UTC m=+1971.311310387" Oct 31 23:23:13.417143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.417074 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podStartSLOduration=2.41705237 podStartE2EDuration="2.41705237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:13.408405128 +0000 UTC m=+1971.391147507" watchObservedRunningTime="2026-10-31 23:23:13.41705237 +0000 UTC m=+1971.399794750" Oct 31 23:23:13.510076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.510034 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:13.998125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:13.998089 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:14.000627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.000589 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.000881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.000850 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:14.215098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.215055 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.215304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.215119 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 31 23:23:14.508334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.508295 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.508914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.508876 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:14.927704 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:14.927618 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:23:14.994962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.994901 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.995452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:14.995412 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:15.220610 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.220536 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:15.221023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.220976 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:15.221839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.221820 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:23:15.511118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.511064 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:15.511713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.511124 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:15.736013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.733271 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:23:15.909835 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.909744 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:23:15.912079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.910157 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:15.918442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:15.918418 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-836c99-openshift-default-dockercfg-zn6fh\"" Oct 31 23:23:16.007014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.005470 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:16.007014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.005534 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:16.010068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010029 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010233 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010092 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010233 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010180 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010238 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010426 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010516 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010638 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47lr\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010691 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.010772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.010739 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.113288 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.113222 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.113851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.113825 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.114006 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.113974 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.114570 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.114545 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47lr\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.114655 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.114634 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.115005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.114972 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.115093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.115075 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.115373 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.115353 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.115467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.115450 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.115924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.115902 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.116031 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.116020 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.116136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.116119 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.116540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.116517 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.119315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.119286 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.119530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.119501 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.121779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.121755 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.138270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.135519 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47lr\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.139078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.138946 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.235186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.235082 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:16.767243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.705145 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:16.767243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:16.705203 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:17.066412 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.039863 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:17.066412 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.039929 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:17.212970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.209999 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:23:17.226401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.226356 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:17.226514 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.226465 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:17.226553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.226523 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:17.478420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.478364 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" event={"ID":"821e4f82-82fa-47a5-b918-c2e782987f36","Type":"ContainerStarted","Data":"f7f2cfbc1f1a516438ffcfe2c6aef73ff8fea1df683183e69985f8f220336b46"} Oct 31 23:23:17.514526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.514487 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:17.514767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.514737 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:17.999527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.999121 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:17.999527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:17.999186 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:18.503071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:18.485236 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:18.503071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:18.485303 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:18.536950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:18.536910 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" event={"ID":"821e4f82-82fa-47a5-b918-c2e782987f36","Type":"ContainerStarted","Data":"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e"} Oct 31 23:23:18.626654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:18.626583 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podStartSLOduration=3.626564053 podStartE2EDuration="3.626564053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:18.624153863 +0000 UTC m=+1976.606896255" watchObservedRunningTime="2026-10-31 23:23:18.626564053 +0000 UTC m=+1976.609306432" Oct 31 23:23:19.002032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.001546 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:23:19.002032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.001610 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 23:23:19.241377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.241344 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:19.241870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.241838 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:19.242021 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.241902 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:19.497072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.482361 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:19.497072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:19.482420 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:20.105916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.105882 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:20.107059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.107032 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:20.108110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.108088 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:23:20.235659 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.235611 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.235862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.235680 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:20.478925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.478820 7074 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.478925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:20.478890 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 23:23:21.236080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:21.236041 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:21.236480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:21.236099 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:21.495739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:21.495637 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:21.529190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:21.529157 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:21.548024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:21.544388 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:22.235952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:22.235918 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:22.236184 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:22.235972 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:22.617276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:22.617239 7074 scope.go:122] "RemoveContainer" containerID="7c257471b7060a622c45eac4961dd4b138fe82057feed974731df1aa08bd20c1" Oct 31 23:23:23.236803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:23.235860 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:23.236803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:23.235921 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:24.235945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:24.235866 7074 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 23:23:24.235945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:24.235922 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 23:23:25.244341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:25.244306 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:25.245770 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:25.245744 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:25.247005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:25.246969 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:23:26.533639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.533603 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:23:26.541455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.541426 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.626250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.625179 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:23:26.834075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.823362 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvpc\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.844174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.823519 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.844597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.844573 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.844958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.844937 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.845526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.845222 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.845793 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.845775 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.845935 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.845919 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.846348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.846059 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.846519 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.846502 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.950719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950684 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.950914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950772 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.950914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950843 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.950914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950868 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.950914 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950894 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.951122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950924 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.951122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.950982 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvpc\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.951230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.951139 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.951278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.951228 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.951561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.951526 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.952260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.952020 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.952396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.952350 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.953033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.952976 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.953297 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.953276 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.954330 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.954305 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.954423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.954306 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.967404 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.967368 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:26.969449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:26.969418 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvpc\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:27.194730 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.194593 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:27.513513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.511816 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:23:27.513513 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:23:27.512747 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f5523b_cb46_4d56_a70d_c8460db757d9.slice/crio-2d59b44d15da00b61e9f938b128cb72b7d69ac76c33b340fed9d16aca850876f WatchSource:0}: Error finding container 2d59b44d15da00b61e9f938b128cb72b7d69ac76c33b340fed9d16aca850876f: Status 404 returned error can't find the container with id 2d59b44d15da00b61e9f938b128cb72b7d69ac76c33b340fed9d16aca850876f Oct 31 23:23:27.516328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.516285 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:27.516440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.516368 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:27.516440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.516401 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:27.924363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.924248 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" event={"ID":"d2f5523b-cb46-4d56-a70d-c8460db757d9","Type":"ContainerStarted","Data":"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad"} Oct 31 23:23:27.924363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.924325 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" event={"ID":"d2f5523b-cb46-4d56-a70d-c8460db757d9","Type":"ContainerStarted","Data":"2d59b44d15da00b61e9f938b128cb72b7d69ac76c33b340fed9d16aca850876f"} Oct 31 23:23:27.964625 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:27.964568 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podStartSLOduration=1.964553935 podStartE2EDuration="1.964553935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:27.959630243 +0000 UTC m=+1985.942372622" watchObservedRunningTime="2026-10-31 23:23:27.964553935 +0000 UTC m=+1985.947296313" Oct 31 23:23:28.195169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:28.195066 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:28.197112 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:28.197080 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:28.197249 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:28.197142 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:29.199480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:29.199442 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:29.199908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:29.199520 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:30.195983 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:30.195941 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:30.196221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:30.196022 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:31.195148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:31.195115 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:31.195563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:31.195169 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:32.195513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:32.195478 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:32.195916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:32.195533 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:33.195165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:33.195127 7074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 23:23:33.195361 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:33.195181 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 23:23:34.200916 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:34.200883 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:34.201540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:34.201514 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:34.202208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:34.202186 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:23:36.998041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:36.996586 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:23:37.003667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.003636 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.019846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.019521 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:23:37.113645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.113601 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.113851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.113671 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.113851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.113765 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.113851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.113820 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.114034 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.114010 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.114156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.114058 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.114156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.114080 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.114236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.114159 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44sc\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.114236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.114187 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216275 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216340 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216376 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216416 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216584 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216565 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216620 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216647 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216703 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44sc\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216740 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.216880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216800 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.217009 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.216969 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.217164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.217142 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.217164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.217150 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.217306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.217165 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.218923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.218892 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.219217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.219195 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.229201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.229164 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.229363 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.229346 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44sc\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.349864 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.349763 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:37.721486 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:23:37.721453 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c7005a_e66f_472e_8dfe_174ff1b66690.slice/crio-0761d3d957217dada94d3738266d2440a47ae929ac4d1aeb0bc1104ea85692dd WatchSource:0}: Error finding container 0761d3d957217dada94d3738266d2440a47ae929ac4d1aeb0bc1104ea85692dd: Status 404 returned error can't find the container with id 0761d3d957217dada94d3738266d2440a47ae929ac4d1aeb0bc1104ea85692dd Oct 31 23:23:37.723178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.723148 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:23:37.725191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.725146 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:37.725276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.725240 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:37.725312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:37.725284 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:23:38.068355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.068310 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" event={"ID":"b9c7005a-e66f-472e-8dfe-174ff1b66690","Type":"ContainerStarted","Data":"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e"} Oct 31 23:23:38.068875 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.068386 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" event={"ID":"b9c7005a-e66f-472e-8dfe-174ff1b66690","Type":"ContainerStarted","Data":"0761d3d957217dada94d3738266d2440a47ae929ac4d1aeb0bc1104ea85692dd"} Oct 31 23:23:38.095381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.095306 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podStartSLOduration=2.095285863 podStartE2EDuration="2.095285863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:38.093474754 +0000 UTC m=+1996.076217143" watchObservedRunningTime="2026-10-31 23:23:38.095285863 +0000 UTC m=+1996.078028242" Oct 31 23:23:38.351579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.351481 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:38.359156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.359114 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:38.359415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:38.359385 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:39.351738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:39.350549 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:39.351738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:39.350614 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:40.356015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:40.353765 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:40.356015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:40.353826 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:41.351023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:41.350431 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:41.351023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:41.350516 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:42.352482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:42.352441 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:42.353121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:42.353085 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:43.358016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:43.352516 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 23:23:43.358016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:43.352630 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 23:23:43.535951 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:43.535913 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:23:44.976010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:44.971489 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:44.976010 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:44.973199 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:44.984569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:44.983824 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:23:45.098101 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:45.096410 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:23:48.553012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:48.546088 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" podUID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" containerName="payload-processing" containerID="cri-o://b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6" gracePeriod=25 Oct 31 23:23:49.037752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.037721 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:23:49.134979 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.134934 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\") pod \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " Oct 31 23:23:49.135391 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.135364 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume" pod "01be9bb2-abc4-402c-84c9-8ea1f9c6d267" (UID: "01be9bb2-abc4-402c-84c9-8ea1f9c6d267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:49.135702 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.135669 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\" (UniqueName: \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\") pod \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\" (UID: \"01be9bb2-abc4-402c-84c9-8ea1f9c6d267\") " Oct 31 23:23:49.137492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.137469 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:49.138420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.138401 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8" pod "01be9bb2-abc4-402c-84c9-8ea1f9c6d267" (UID: "01be9bb2-abc4-402c-84c9-8ea1f9c6d267"). InnerVolumeSpecName "kube-api-access-8z9s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:49.230620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.230588 7074 generic.go:356] "Generic (PLEG): container finished" podID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" containerID="b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6" exitCode=0 Oct 31 23:23:49.230810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.230635 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" Oct 31 23:23:49.230810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.230671 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" event={"ID":"01be9bb2-abc4-402c-84c9-8ea1f9c6d267","Type":"ContainerDied","Data":"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6"} Oct 31 23:23:49.230810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.230731 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq" event={"ID":"01be9bb2-abc4-402c-84c9-8ea1f9c6d267","Type":"ContainerDied","Data":"257cfe135e7afdbe6a58202edef69c87b80cf9208aa7806d0afb077f0e9d2ff1"} Oct 31 23:23:49.230810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.230756 7074 scope.go:122] "RemoveContainer" containerID="b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6" Oct 31 23:23:49.239876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.239735 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9s8\" (UniqueName: \"kubernetes.io/projected/01be9bb2-abc4-402c-84c9-8ea1f9c6d267-kube-api-access-8z9s8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:49.248459 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.248434 7074 scope.go:122] "RemoveContainer" containerID="b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6" Oct 31 23:23:49.251389 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:49.249533 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6\": container with ID starting with b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6 not found: ID does not exist" containerID="b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6" Oct 31 23:23:49.251389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.249577 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6"} err="failed to get container status \"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6\": rpc error: code = NotFound desc = could not find container \"b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6\": container with ID starting with b8f026762102c38d8e0733ba2fe39a1aad4c0e4b1dcac4dbf2dba96955c4e8b6 not found: ID does not exist" Oct 31 23:23:49.260785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.260749 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:23:49.265060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:49.265031 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6a08-5d6cdb6b8-sw5bq"] Oct 31 23:23:50.276015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:50.275308 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" path="/var/lib/kubelet/pods/01be9bb2-abc4-402c-84c9-8ea1f9c6d267/volumes" Oct 31 23:23:53.902232 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:53.902198 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:23:53.902696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:53.902470 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" containerID="cri-o://0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a" gracePeriod=30 Oct 31 23:23:53.908639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:53.908604 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:23:53.908874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:53.908847 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" containerID="cri-o://3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f" gracePeriod=30 Oct 31 23:23:56.675838 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:56.674863 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:23:56.675838 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:56.675245 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerName="maas-api" containerID="cri-o://cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176" gracePeriod=30 Oct 31 23:23:56.735072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:56.733890 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.106:8443/health\": dial tcp 10.134.0.106:8443: connect: connection refused" Oct 31 23:23:57.140320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.140290 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:23:57.234968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.234877 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\") pod \"7d0ac0ce-89d2-4837-a771-ba4636017709\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " Oct 31 23:23:57.235188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.235166 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\" (UniqueName: \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\") pod \"7d0ac0ce-89d2-4837-a771-ba4636017709\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " Oct 31 23:23:57.235257 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.235222 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\") pod \"7d0ac0ce-89d2-4837-a771-ba4636017709\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " Oct 31 23:23:57.235331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.235258 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") pod \"7d0ac0ce-89d2-4837-a771-ba4636017709\" (UID: \"7d0ac0ce-89d2-4837-a771-ba4636017709\") " Oct 31 23:23:57.237541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.237510 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca" pod "7d0ac0ce-89d2-4837-a771-ba4636017709" (UID: "7d0ac0ce-89d2-4837-a771-ba4636017709"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:57.239946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.239919 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs" pod "7d0ac0ce-89d2-4837-a771-ba4636017709" (UID: "7d0ac0ce-89d2-4837-a771-ba4636017709"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:57.241569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.241354 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls" pod "7d0ac0ce-89d2-4837-a771-ba4636017709" (UID: "7d0ac0ce-89d2-4837-a771-ba4636017709"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:57.246321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.246297 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf" pod "7d0ac0ce-89d2-4837-a771-ba4636017709" (UID: "7d0ac0ce-89d2-4837-a771-ba4636017709"). InnerVolumeSpecName "kube-api-access-p9lzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:57.337968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.337923 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.337968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.337957 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lzf\" (UniqueName: \"kubernetes.io/projected/7d0ac0ce-89d2-4837-a771-ba4636017709-kube-api-access-p9lzf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.337968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.337973 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0ac0ce-89d2-4837-a771-ba4636017709-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.338251 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.337985 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0ac0ce-89d2-4837-a771-ba4636017709-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.369617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.369566 7074 generic.go:356] "Generic (PLEG): container finished" podID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerID="cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176" exitCode=0 Oct 31 23:23:57.369793 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.369656 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" Oct 31 23:23:57.369847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.369801 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" event={"ID":"7d0ac0ce-89d2-4837-a771-ba4636017709","Type":"ContainerDied","Data":"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176"} Oct 31 23:23:57.369892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.369855 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp" event={"ID":"7d0ac0ce-89d2-4837-a771-ba4636017709","Type":"ContainerDied","Data":"7a8d17a47820dfd6612c1037e9db16b0f78636fe94cfd940342b94098cacfd53"} Oct 31 23:23:57.369892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.369888 7074 scope.go:122] "RemoveContainer" containerID="cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176" Oct 31 23:23:57.386202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.386179 7074 scope.go:122] "RemoveContainer" containerID="cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176" Oct 31 23:23:57.391779 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:57.391742 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176\": container with ID starting with cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176 not found: ID does not exist" containerID="cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176" Oct 31 23:23:57.391892 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.391791 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176"} err="failed to get container status \"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176\": rpc error: code = NotFound desc = could not find container \"cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176\": container with ID starting with cc543d20dd30b68e070405f2887b6ab2d138496c6f0ea43ad6de8937d074b176 not found: ID does not exist" Oct 31 23:23:57.405318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.405286 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:23:57.412946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:57.412922 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8e6a08-555d74f6db-8xjhp"] Oct 31 23:23:58.147650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:58.147605 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" path="/var/lib/kubelet/pods/7d0ac0ce-89d2-4837-a771-ba4636017709/volumes" Oct 31 23:23:59.228593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.228563 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:23:59.275633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.275600 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.275784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.275695 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.275894 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.275879 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.275963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.275907 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.276049 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.275965 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.276049 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.276013 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.276148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.276057 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.276205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.276181 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.276383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.276357 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.276491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.276402 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.281014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.277295 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.281014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.278873 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\") pod \"105e3e1e-8500-4de6-b605-a5f79adb683a\" (UID: \"105e3e1e-8500-4de6-b605-a5f79adb683a\") " Oct 31 23:23:59.281014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.279598 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:59.281014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.280972 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.281667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.281627 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "kube-api-access-7lhnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.283658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.283634 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:59.286426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.286403 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:59.293218 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.293195 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105e3e1e-8500-4de6-b605-a5f79adb683a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.293309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.293225 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.293309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.293241 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.293309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.293258 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.293309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.293273 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.294899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.294874 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.295174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.295146 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token" pod "105e3e1e-8500-4de6-b605-a5f79adb683a" (UID: "105e3e1e-8500-4de6-b605-a5f79adb683a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.393847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.393809 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.393960 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394121 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394140 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.394217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394163 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394198 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394369 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394233 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394369 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394284 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.394517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394476 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394615 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394664 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394671 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\") pod \"13751078-df8a-4c5d-a011-4bf0edc29cb9\" (UID: \"13751078-df8a-4c5d-a011-4bf0edc29cb9\") " Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394707 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.394853 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395027 7074 generic.go:356] "Generic (PLEG): container finished" podID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerID="0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a" exitCode=0 Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395102 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395122 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" event={"ID":"105e3e1e-8500-4de6-b605-a5f79adb683a","Type":"ContainerDied","Data":"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a"} Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395183 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks" event={"ID":"105e3e1e-8500-4de6-b605-a5f79adb683a","Type":"ContainerDied","Data":"56b43083548536e9c3b2acc4c05e580502a7c906a6cb57fde10b69affc0baf73"} Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395216 7074 scope.go:122] "RemoveContainer" containerID="0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395648 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395670 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395686 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395701 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395718 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395733 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13751078-df8a-4c5d-a011-4bf0edc29cb9-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395747 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395760 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.395927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.395777 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhnj\" (UniqueName: \"kubernetes.io/projected/105e3e1e-8500-4de6-b605-a5f79adb683a-kube-api-access-7lhnj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.397690 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "kube-api-access-5725h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.397921 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.398714 7074 generic.go:356] "Generic (PLEG): container finished" podID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerID="3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f" exitCode=0 Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.398952 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" event={"ID":"13751078-df8a-4c5d-a011-4bf0edc29cb9","Type":"ContainerDied","Data":"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f"} Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.399082 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" event={"ID":"13751078-df8a-4c5d-a011-4bf0edc29cb9","Type":"ContainerDied","Data":"409fc5d45f62d2872ed8f09ff0032b097c246cc7f819b14143c30c63459424aa"} Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.399178 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9" Oct 31 23:23:59.399868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.399301 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.403819 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.403777 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo" pod "13751078-df8a-4c5d-a011-4bf0edc29cb9" (UID: "13751078-df8a-4c5d-a011-4bf0edc29cb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:59.419435 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.419376 7074 scope.go:122] "RemoveContainer" containerID="0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a" Oct 31 23:23:59.421853 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:59.421781 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a\": container with ID starting with 0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a not found: ID does not exist" containerID="0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a" Oct 31 23:23:59.422039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.422012 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a"} err="failed to get container status \"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a\": rpc error: code = NotFound desc = could not find container \"0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a\": container with ID starting with 0b63730728afddee84cc099eba4820e8f4f9da796f8e75dc8deb389f20e4a49a not found: ID does not exist" Oct 31 23:23:59.422156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.422143 7074 scope.go:122] "RemoveContainer" containerID="3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f" Oct 31 23:23:59.448711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.448635 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:23:59.454279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.454251 7074 scope.go:122] "RemoveContainer" containerID="3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f" Oct 31 23:23:59.454806 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:23:59.454778 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f\": container with ID starting with 3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f not found: ID does not exist" containerID="3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f" Oct 31 23:23:59.454973 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.454944 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f"} err="failed to get container status \"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f\": rpc error: code = NotFound desc = could not find container \"3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f\": container with ID starting with 3236122e717ce49b670591ddd2c755ce696b5fa8e8cccd18e0947928f008921f not found: ID does not exist" Oct 31 23:23:59.455376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.455348 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-qswks"] Oct 31 23:23:59.498087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.498054 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5725h\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-kube-api-access-5725h\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.498087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.498081 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.498087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.498091 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.498297 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.498099 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13751078-df8a-4c5d-a011-4bf0edc29cb9-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.731133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.731096 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:23:59.738227 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:23:59.738196 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k5ml9"] Oct 31 23:24:00.146843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:00.146809 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" path="/var/lib/kubelet/pods/105e3e1e-8500-4de6-b605-a5f79adb683a/volumes" Oct 31 23:24:00.147434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:00.147418 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" path="/var/lib/kubelet/pods/13751078-df8a-4c5d-a011-4bf0edc29cb9/volumes" Oct 31 23:24:05.670015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:05.655424 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:24:05.670015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:05.655769 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" containerID="cri-o://b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392" gracePeriod=30 Oct 31 23:24:05.670015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:05.669904 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:24:05.670879 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:05.670164 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" containerID="cri-o://fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b" gracePeriod=30 Oct 31 23:24:09.856048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:09.855957 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:24:09.856572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:09.856247 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" podUID="3fc753cb-a030-47bc-a64d-dee62ff222b0" containerName="maas-api" containerID="cri-o://244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54" gracePeriod=30 Oct 31 23:24:10.393517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.393486 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:24:10.439019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.438930 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\") pod \"3fc753cb-a030-47bc-a64d-dee62ff222b0\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " Oct 31 23:24:10.439174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.439034 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") pod \"3fc753cb-a030-47bc-a64d-dee62ff222b0\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " Oct 31 23:24:10.439232 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.439156 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\" (UniqueName: \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\") pod \"3fc753cb-a030-47bc-a64d-dee62ff222b0\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " Oct 31 23:24:10.439280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.439242 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\") pod \"3fc753cb-a030-47bc-a64d-dee62ff222b0\" (UID: \"3fc753cb-a030-47bc-a64d-dee62ff222b0\") " Oct 31 23:24:10.439460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.439425 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca" pod "3fc753cb-a030-47bc-a64d-dee62ff222b0" (UID: "3fc753cb-a030-47bc-a64d-dee62ff222b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:10.440139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.440116 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc753cb-a030-47bc-a64d-dee62ff222b0-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.442125 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.442104 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls" pod "3fc753cb-a030-47bc-a64d-dee62ff222b0" (UID: "3fc753cb-a030-47bc-a64d-dee62ff222b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:10.442268 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.442242 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc" pod "3fc753cb-a030-47bc-a64d-dee62ff222b0" (UID: "3fc753cb-a030-47bc-a64d-dee62ff222b0"). InnerVolumeSpecName "kube-api-access-mx9hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:10.442402 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.442379 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs" pod "3fc753cb-a030-47bc-a64d-dee62ff222b0" (UID: "3fc753cb-a030-47bc-a64d-dee62ff222b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:10.566053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.548379 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.566053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.557266 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9hc\" (UniqueName: \"kubernetes.io/projected/3fc753cb-a030-47bc-a64d-dee62ff222b0-kube-api-access-mx9hc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.566053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.557298 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc753cb-a030-47bc-a64d-dee62ff222b0-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.576861 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.576829 7074 generic.go:356] "Generic (PLEG): container finished" podID="3fc753cb-a030-47bc-a64d-dee62ff222b0" containerID="244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54" exitCode=0 Oct 31 23:24:10.577103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.576873 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" Oct 31 23:24:10.577103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.576906 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" event={"ID":"3fc753cb-a030-47bc-a64d-dee62ff222b0","Type":"ContainerDied","Data":"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54"} Oct 31 23:24:10.577103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.576964 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj" event={"ID":"3fc753cb-a030-47bc-a64d-dee62ff222b0","Type":"ContainerDied","Data":"424b1177ec9c7a21129ea4e4d8214b64bc1a28f47f2b3711941e76aec85486ac"} Oct 31 23:24:10.577103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.577000 7074 scope.go:122] "RemoveContainer" containerID="244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54" Oct 31 23:24:10.596492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.596467 7074 scope.go:122] "RemoveContainer" containerID="244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54" Oct 31 23:24:10.596860 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:10.596830 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54\": container with ID starting with 244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54 not found: ID does not exist" containerID="244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54" Oct 31 23:24:10.596970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.596863 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54"} err="failed to get container status \"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54\": rpc error: code = NotFound desc = could not find container \"244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54\": container with ID starting with 244db4c75df1cb2c2c5ff782f796cd7d1463b4fec94a46d1abdff9a86fa3aa54 not found: ID does not exist" Oct 31 23:24:10.605641 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.605586 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:24:10.611166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.611143 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2126ff-9ff9bf99d-5k7vj"] Oct 31 23:24:10.967748 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:10.967689 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:24:11.008690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.008660 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:24:11.064842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.064810 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.064878 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.065060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.064928 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.065060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065038 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065208 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.065302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065272 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.065777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065554 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065608 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065636 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065689 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.065777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.065739 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.066549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.066202 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.066549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.066393 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.066549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.066458 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.066720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.066682 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.066776 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.066739 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.067110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.067087 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.067370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.067345 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.067539 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.067519 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.067959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.067934 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.068424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068395 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.068526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068475 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.068526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068509 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\") pod \"722083a0-8d6b-40e7-be1d-011aaa3df44e\" (UID: \"722083a0-8d6b-40e7-be1d-011aaa3df44e\") " Oct 31 23:24:11.068640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068552 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.068640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068627 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\") pod \"47a55d87-5ecd-4ffb-a601-701e66281d43\" (UID: \"47a55d87-5ecd-4ffb-a601-701e66281d43\") " Oct 31 23:24:11.068835 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.068807 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.069069 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069045 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.069290 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069269 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.069751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069725 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.069751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069732 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a55d87-5ecd-4ffb-a601-701e66281d43-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069759 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069781 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069833 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069852 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069850 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069866 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069883 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069897 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069910 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722083a0-8d6b-40e7-be1d-011aaa3df44e-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.069927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069925 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.070271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.069939 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.070271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.070186 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.070271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.070205 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "kube-api-access-tdwf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.071343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.071317 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:11.071632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.071612 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo" pod "722083a0-8d6b-40e7-be1d-011aaa3df44e" (UID: "722083a0-8d6b-40e7-be1d-011aaa3df44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:11.071918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.071903 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9" pod "47a55d87-5ecd-4ffb-a601-701e66281d43" (UID: "47a55d87-5ecd-4ffb-a601-701e66281d43"). InnerVolumeSpecName "kube-api-access-sbzt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172752 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172776 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172786 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172799 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbzt9\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-kube-api-access-sbzt9\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172810 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a55d87-5ecd-4ffb-a601-701e66281d43-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172819 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172828 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwf4\" (UniqueName: \"kubernetes.io/projected/722083a0-8d6b-40e7-be1d-011aaa3df44e-kube-api-access-tdwf4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.172841 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.172840 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722083a0-8d6b-40e7-be1d-011aaa3df44e-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.589893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.589859 7074 generic.go:356] "Generic (PLEG): container finished" podID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerID="b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392" exitCode=0 Oct 31 23:24:11.590154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.589938 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" Oct 31 23:24:11.590154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.589945 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" event={"ID":"47a55d87-5ecd-4ffb-a601-701e66281d43","Type":"ContainerDied","Data":"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392"} Oct 31 23:24:11.590154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.590086 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5" event={"ID":"47a55d87-5ecd-4ffb-a601-701e66281d43","Type":"ContainerDied","Data":"2ae14dec16d738e8c0588e5cfd42182a5de13a41a7dd1ff4ff0aade053534bd5"} Oct 31 23:24:11.590154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.590127 7074 scope.go:122] "RemoveContainer" containerID="b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392" Oct 31 23:24:11.591844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.591820 7074 generic.go:356] "Generic (PLEG): container finished" podID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerID="fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b" exitCode=0 Oct 31 23:24:11.591962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.591914 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" Oct 31 23:24:11.592089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.591975 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" event={"ID":"722083a0-8d6b-40e7-be1d-011aaa3df44e","Type":"ContainerDied","Data":"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b"} Oct 31 23:24:11.592089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.592036 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9" event={"ID":"722083a0-8d6b-40e7-be1d-011aaa3df44e","Type":"ContainerDied","Data":"f57a6fc0bfb72c15470114e98a9c04cfdd38dc707380eed79dd94489d80b008e"} Oct 31 23:24:11.603497 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.603479 7074 scope.go:122] "RemoveContainer" containerID="b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392" Oct 31 23:24:11.603922 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:11.603825 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392\": container with ID starting with b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392 not found: ID does not exist" containerID="b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392" Oct 31 23:24:11.603922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.603860 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392"} err="failed to get container status \"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392\": rpc error: code = NotFound desc = could not find container \"b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392\": container with ID starting with b4125a9c786323034a572107f51f78b3a2373b6a047d87f9ec446821652c6392 not found: ID does not exist" Oct 31 23:24:11.603922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.603883 7074 scope.go:122] "RemoveContainer" containerID="fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b" Oct 31 23:24:11.617454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.617340 7074 scope.go:122] "RemoveContainer" containerID="fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b" Oct 31 23:24:11.618291 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:11.618267 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b\": container with ID starting with fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b not found: ID does not exist" containerID="fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b" Oct 31 23:24:11.618384 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.618295 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b"} err="failed to get container status \"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b\": rpc error: code = NotFound desc = could not find container \"fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b\": container with ID starting with fd06120acd29e41cd86725a3ec0f9bdfee6f2b332104e24eb664737f8725353b not found: ID does not exist" Oct 31 23:24:11.622862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.622839 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:24:11.627214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.627195 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-bspl9"] Oct 31 23:24:11.639605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.639578 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:24:11.643461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:11.643441 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-rkls5"] Oct 31 23:24:12.148104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:12.148025 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc753cb-a030-47bc-a64d-dee62ff222b0" path="/var/lib/kubelet/pods/3fc753cb-a030-47bc-a64d-dee62ff222b0/volumes" Oct 31 23:24:12.148876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:12.148853 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" path="/var/lib/kubelet/pods/47a55d87-5ecd-4ffb-a601-701e66281d43/volumes" Oct 31 23:24:12.149516 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:12.149495 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" path="/var/lib/kubelet/pods/722083a0-8d6b-40e7-be1d-011aaa3df44e/volumes" Oct 31 23:24:13.227678 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:13.227643 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:24:13.228161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:13.227936 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" containerID="cri-o://9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e" gracePeriod=30 Oct 31 23:24:13.234193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:13.234164 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:24:13.234429 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:13.234404 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" containerID="cri-o://bac2ac4e4c0b373bfffbabe9aece95a1e28b3ac86248021128955dd40b2857a5" gracePeriod=30 Oct 31 23:24:14.893095 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:14.893050 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:24:14.975341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:14.975301 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:14.975513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:14.975378 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:15.223856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:15.223774 7074 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:15.223856 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:15.223840 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:17.400019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.399036 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:24:17.400019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.399734 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" podUID="2a8ce167-55ad-47ac-9cee-731114a42433" containerName="maas-api" containerID="cri-o://598f0188877aceb8556f10421f813648003e225109be621b747c224cf948ff84" gracePeriod=30 Oct 31 23:24:17.693471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.693439 7074 generic.go:356] "Generic (PLEG): container finished" podID="2a8ce167-55ad-47ac-9cee-731114a42433" containerID="598f0188877aceb8556f10421f813648003e225109be621b747c224cf948ff84" exitCode=0 Oct 31 23:24:17.693627 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.693516 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" event={"ID":"2a8ce167-55ad-47ac-9cee-731114a42433","Type":"ContainerDied","Data":"598f0188877aceb8556f10421f813648003e225109be621b747c224cf948ff84"} Oct 31 23:24:17.731194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.731165 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:24:17.777881 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.777842 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\" (UniqueName: \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\") pod \"2a8ce167-55ad-47ac-9cee-731114a42433\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " Oct 31 23:24:17.778187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.777906 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\") pod \"2a8ce167-55ad-47ac-9cee-731114a42433\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " Oct 31 23:24:17.778187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.778042 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\") pod \"2a8ce167-55ad-47ac-9cee-731114a42433\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " Oct 31 23:24:17.778187 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.778115 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\") pod \"2a8ce167-55ad-47ac-9cee-731114a42433\" (UID: \"2a8ce167-55ad-47ac-9cee-731114a42433\") " Oct 31 23:24:17.779900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.779865 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca" pod "2a8ce167-55ad-47ac-9cee-731114a42433" (UID: "2a8ce167-55ad-47ac-9cee-731114a42433"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:17.783104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.783068 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp" pod "2a8ce167-55ad-47ac-9cee-731114a42433" (UID: "2a8ce167-55ad-47ac-9cee-731114a42433"). InnerVolumeSpecName "kube-api-access-md6dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:17.784144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.784113 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls" pod "2a8ce167-55ad-47ac-9cee-731114a42433" (UID: "2a8ce167-55ad-47ac-9cee-731114a42433"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:17.794217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.794170 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs" pod "2a8ce167-55ad-47ac-9cee-731114a42433" (UID: "2a8ce167-55ad-47ac-9cee-731114a42433"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:17.920014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.919700 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6dp\" (UniqueName: \"kubernetes.io/projected/2a8ce167-55ad-47ac-9cee-731114a42433-kube-api-access-md6dp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.920014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.919744 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ce167-55ad-47ac-9cee-731114a42433-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.920014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.919761 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.920014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:17.919776 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8ce167-55ad-47ac-9cee-731114a42433-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.668144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.668110 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:24:18.721805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.721771 7074 generic.go:356] "Generic (PLEG): container finished" podID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerID="9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e" exitCode=0 Oct 31 23:24:18.722086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.721831 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" Oct 31 23:24:18.722086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.721862 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" event={"ID":"b9c7005a-e66f-472e-8dfe-174ff1b66690","Type":"ContainerDied","Data":"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e"} Oct 31 23:24:18.722086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.721921 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl" event={"ID":"b9c7005a-e66f-472e-8dfe-174ff1b66690","Type":"ContainerDied","Data":"0761d3d957217dada94d3738266d2440a47ae929ac4d1aeb0bc1104ea85692dd"} Oct 31 23:24:18.722086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.721956 7074 scope.go:122] "RemoveContainer" containerID="9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e" Oct 31 23:24:18.723749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.723657 7074 generic.go:356] "Generic (PLEG): container finished" podID="80e23523-c883-45b6-bd79-a5062bf91962" containerID="bac2ac4e4c0b373bfffbabe9aece95a1e28b3ac86248021128955dd40b2857a5" exitCode=0 Oct 31 23:24:18.723852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.723831 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" event={"ID":"80e23523-c883-45b6-bd79-a5062bf91962","Type":"ContainerDied","Data":"bac2ac4e4c0b373bfffbabe9aece95a1e28b3ac86248021128955dd40b2857a5"} Oct 31 23:24:18.726343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.726315 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" event={"ID":"2a8ce167-55ad-47ac-9cee-731114a42433","Type":"ContainerDied","Data":"d71bc4187b5240ad01884323685e80c1533c77ffb50b1c7621a4e5500a6245f3"} Oct 31 23:24:18.726471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.726411 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c" Oct 31 23:24:18.752831 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.752810 7074 scope.go:122] "RemoveContainer" containerID="9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e" Oct 31 23:24:18.753280 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:18.753150 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e\": container with ID starting with 9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e not found: ID does not exist" containerID="9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e" Oct 31 23:24:18.753280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.753187 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e"} err="failed to get container status \"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e\": rpc error: code = NotFound desc = could not find container \"9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e\": container with ID starting with 9ddeb0f3ad67f0742f8862357c591abf2cb38582697b7f19158203d7bc098b2e not found: ID does not exist" Oct 31 23:24:18.753280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.753209 7074 scope.go:122] "RemoveContainer" containerID="598f0188877aceb8556f10421f813648003e225109be621b747c224cf948ff84" Oct 31 23:24:18.757488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.757464 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:24:18.772886 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.772854 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:24:18.781441 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.781406 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f80dfa-5669bc8dd6-6mf2c"] Oct 31 23:24:18.860492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860448 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.860694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860601 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.860694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860642 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.860694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860679 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.860866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860709 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.860866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860743 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.860866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860858 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860901 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860912 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.860931 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861019 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861062 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.861085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861078 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861072 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861171 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861248 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861277 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861306 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.861390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861370 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:18.861706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861395 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861440 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.861706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861480 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\") pod \"80e23523-c883-45b6-bd79-a5062bf91962\" (UID: \"80e23523-c883-45b6-bd79-a5062bf91962\") " Oct 31 23:24:18.861706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861601 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\") pod \"b9c7005a-e66f-472e-8dfe-174ff1b66690\" (UID: \"b9c7005a-e66f-472e-8dfe-174ff1b66690\") " Oct 31 23:24:18.861706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.861610 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.862497 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862466 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c7005a-e66f-472e-8dfe-174ff1b66690-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.862612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862502 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.862612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862517 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.862612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862530 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.862612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862543 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.862612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.862556 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e23523-c883-45b6-bd79-a5062bf91962-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.863081 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.863059 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.865044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.864981 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.865044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865018 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:18.865044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865021 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:18.865334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865197 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "kube-api-access-q44sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:18.865334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865244 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.865334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865264 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.865334 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865278 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:18.865540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865338 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.865540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865425 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n" pod "80e23523-c883-45b6-bd79-a5062bf91962" (UID: "80e23523-c883-45b6-bd79-a5062bf91962"). InnerVolumeSpecName "kube-api-access-dzz9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:18.865656 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865635 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:18.865790 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.865771 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy" pod "b9c7005a-e66f-472e-8dfe-174ff1b66690" (UID: "b9c7005a-e66f-472e-8dfe-174ff1b66690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964128 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzz9n\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-kube-api-access-dzz9n\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964163 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964173 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e23523-c883-45b6-bd79-a5062bf91962-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964182 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44sc\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-kube-api-access-q44sc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964192 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964201 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964209 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964216 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964217 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e23523-c883-45b6-bd79-a5062bf91962-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964226 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964235 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e23523-c883-45b6-bd79-a5062bf91962-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964243 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:18.964593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:18.964251 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c7005a-e66f-472e-8dfe-174ff1b66690-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.061652 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.061594 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:24:19.073555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.073510 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-g6qtl"] Oct 31 23:24:19.745234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.745139 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" event={"ID":"80e23523-c883-45b6-bd79-a5062bf91962","Type":"ContainerDied","Data":"1c21df2bef3c336258e857016dbe15bf0eb73c1fcb692e485099dedc86a369ca"} Oct 31 23:24:19.745234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.745172 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p" Oct 31 23:24:19.745234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.745212 7074 scope.go:122] "RemoveContainer" containerID="bac2ac4e4c0b373bfffbabe9aece95a1e28b3ac86248021128955dd40b2857a5" Oct 31 23:24:19.773454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.773416 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:24:19.777672 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:19.777631 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-vrb6p"] Oct 31 23:24:20.145360 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:20.145329 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8ce167-55ad-47ac-9cee-731114a42433" path="/var/lib/kubelet/pods/2a8ce167-55ad-47ac-9cee-731114a42433/volumes" Oct 31 23:24:20.145889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:20.145875 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e23523-c883-45b6-bd79-a5062bf91962" path="/var/lib/kubelet/pods/80e23523-c883-45b6-bd79-a5062bf91962/volumes" Oct 31 23:24:20.146395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:20.146381 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" path="/var/lib/kubelet/pods/b9c7005a-e66f-472e-8dfe-174ff1b66690/volumes" Oct 31 23:24:21.027608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.027574 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:24:21.028274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.027819 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" podUID="592e575f-eeea-4d6e-bf00-8e258d9f7929" containerName="maas-api" containerID="cri-o://f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6" gracePeriod=30 Oct 31 23:24:21.530764 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.530734 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:24:21.575286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.575251 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\") pod \"592e575f-eeea-4d6e-bf00-8e258d9f7929\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " Oct 31 23:24:21.575457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.575295 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\" (UniqueName: \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\") pod \"592e575f-eeea-4d6e-bf00-8e258d9f7929\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " Oct 31 23:24:21.575457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.575360 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\") pod \"592e575f-eeea-4d6e-bf00-8e258d9f7929\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " Oct 31 23:24:21.575457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.575436 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\") pod \"592e575f-eeea-4d6e-bf00-8e258d9f7929\" (UID: \"592e575f-eeea-4d6e-bf00-8e258d9f7929\") " Oct 31 23:24:21.576480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.576446 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca" pod "592e575f-eeea-4d6e-bf00-8e258d9f7929" (UID: "592e575f-eeea-4d6e-bf00-8e258d9f7929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:21.578354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.578325 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls" pod "592e575f-eeea-4d6e-bf00-8e258d9f7929" (UID: "592e575f-eeea-4d6e-bf00-8e258d9f7929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:21.578604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.578574 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs" pod "592e575f-eeea-4d6e-bf00-8e258d9f7929" (UID: "592e575f-eeea-4d6e-bf00-8e258d9f7929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:21.578701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.578596 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq" pod "592e575f-eeea-4d6e-bf00-8e258d9f7929" (UID: "592e575f-eeea-4d6e-bf00-8e258d9f7929"). InnerVolumeSpecName "kube-api-access-r96lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:21.704320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.704279 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:21.704320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.704317 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96lq\" (UniqueName: \"kubernetes.io/projected/592e575f-eeea-4d6e-bf00-8e258d9f7929-kube-api-access-r96lq\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:21.704595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.704335 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/592e575f-eeea-4d6e-bf00-8e258d9f7929-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:21.705423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.704349 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/592e575f-eeea-4d6e-bf00-8e258d9f7929-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:21.770761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.770716 7074 generic.go:356] "Generic (PLEG): container finished" podID="592e575f-eeea-4d6e-bf00-8e258d9f7929" containerID="f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6" exitCode=0 Oct 31 23:24:21.770972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.770759 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" event={"ID":"592e575f-eeea-4d6e-bf00-8e258d9f7929","Type":"ContainerDied","Data":"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6"} Oct 31 23:24:21.770972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.770880 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" Oct 31 23:24:21.770972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.770923 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm" event={"ID":"592e575f-eeea-4d6e-bf00-8e258d9f7929","Type":"ContainerDied","Data":"9834f1b949faf27435eb2851b871aeaacac818faa14ee5f51f9fb7a6793ec620"} Oct 31 23:24:21.770972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.770948 7074 scope.go:122] "RemoveContainer" containerID="f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6" Oct 31 23:24:21.796588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.796557 7074 scope.go:122] "RemoveContainer" containerID="f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6" Oct 31 23:24:21.797165 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:21.797132 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6\": container with ID starting with f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6 not found: ID does not exist" containerID="f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6" Oct 31 23:24:21.797265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.797176 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6"} err="failed to get container status \"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6\": rpc error: code = NotFound desc = could not find container \"f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6\": container with ID starting with f4eab2d3ecc628b54365d5edc04f172f093dddf8aaef3a0514a146884745c3a6 not found: ID does not exist" Oct 31 23:24:21.809893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.809864 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:24:21.814086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:21.814055 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6685e-db66b4d7f-dhshm"] Oct 31 23:24:22.145067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:22.144955 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592e575f-eeea-4d6e-bf00-8e258d9f7929" path="/var/lib/kubelet/pods/592e575f-eeea-4d6e-bf00-8e258d9f7929/volumes" Oct 31 23:24:27.695275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:27.695236 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:24:27.695731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:27.695439 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" containerID="cri-o://fb09f5a125f3f7a11aee172da8386956ebaf99f6b4455fea8def1fedcc60da3d" gracePeriod=30 Oct 31 23:24:27.789365 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:27.789326 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:24:27.789623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:27.789597 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" containerID="cri-o://2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975" gracePeriod=30 Oct 31 23:24:29.089014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:29.084585 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:24:29.089014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:29.084916 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" containerID="cri-o://b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e" gracePeriod=30 Oct 31 23:24:32.197633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:32.197592 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:32.198294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:32.198255 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:32.892222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:32.892098 7074 generic.go:356] "Generic (PLEG): container finished" podID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerID="fb09f5a125f3f7a11aee172da8386956ebaf99f6b4455fea8def1fedcc60da3d" exitCode=0 Oct 31 23:24:32.892365 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:32.892229 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" event={"ID":"0825b221-898d-4dcc-bdef-fa0f8673a2c8","Type":"ContainerDied","Data":"fb09f5a125f3f7a11aee172da8386956ebaf99f6b4455fea8def1fedcc60da3d"} Oct 31 23:24:33.091925 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.091896 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:24:33.117536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.117489 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:24:33.249407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249365 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249424 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249472 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249508 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249590 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249628 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249656 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249831 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.249910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.249844 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.251399 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.251465 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.252078 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.255240 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.255348 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.255494 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256177 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256736 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256815 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256891 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256924 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.256975 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\") pod \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\" (UID: \"0825b221-898d-4dcc-bdef-fa0f8673a2c8\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.259574 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "kube-api-access-2676f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.259813 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.259956 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.260026 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.260160 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\") pod \"15558e00-bb89-4cac-9cd4-866a5f89d589\" (UID: \"15558e00-bb89-4cac-9cd4-866a5f89d589\") " Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.261968 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.262483 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:33.262846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.262730 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263317 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2676f\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-kube-api-access-2676f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263339 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263355 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263369 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263384 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263398 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263412 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263427 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263440 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263453 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15558e00-bb89-4cac-9cd4-866a5f89d589-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.263467 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.264976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.264950 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.266124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.265260 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:33.266124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.265298 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:33.266124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.265679 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "kube-api-access-p8hxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:33.266417 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.266393 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.267110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.267083 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo" pod "0825b221-898d-4dcc-bdef-fa0f8673a2c8" (UID: "0825b221-898d-4dcc-bdef-fa0f8673a2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:33.267549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.267535 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy" pod "15558e00-bb89-4cac-9cd4-866a5f89d589" (UID: "15558e00-bb89-4cac-9cd4-866a5f89d589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:33.365271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365215 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365259 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365278 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365295 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0825b221-898d-4dcc-bdef-fa0f8673a2c8-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365333 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hxj\" (UniqueName: \"kubernetes.io/projected/0825b221-898d-4dcc-bdef-fa0f8673a2c8-kube-api-access-p8hxj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365351 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.365593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.365365 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15558e00-bb89-4cac-9cd4-866a5f89d589-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.903033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.902977 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" event={"ID":"0825b221-898d-4dcc-bdef-fa0f8673a2c8","Type":"ContainerDied","Data":"acc0a4a67fd71b22d55079472c9358b11dc536121a85b6bebc73cb0b9ec2b6cd"} Oct 31 23:24:33.903033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.902978 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv" Oct 31 23:24:33.903340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.903084 7074 scope.go:122] "RemoveContainer" containerID="fb09f5a125f3f7a11aee172da8386956ebaf99f6b4455fea8def1fedcc60da3d" Oct 31 23:24:33.905205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.905175 7074 generic.go:356] "Generic (PLEG): container finished" podID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerID="2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975" exitCode=0 Oct 31 23:24:33.905327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.905221 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" Oct 31 23:24:33.905327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.905253 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" event={"ID":"15558e00-bb89-4cac-9cd4-866a5f89d589","Type":"ContainerDied","Data":"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975"} Oct 31 23:24:33.905327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.905301 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk" event={"ID":"15558e00-bb89-4cac-9cd4-866a5f89d589","Type":"ContainerDied","Data":"34489b740629b36a435efbc362f0f4ae989817efc16683bbfd8fe58efbdf3262"} Oct 31 23:24:33.915310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.915286 7074 scope.go:122] "RemoveContainer" containerID="2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975" Oct 31 23:24:33.926974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.926952 7074 scope.go:122] "RemoveContainer" containerID="2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975" Oct 31 23:24:33.927390 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:33.927365 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975\": container with ID starting with 2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975 not found: ID does not exist" containerID="2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975" Oct 31 23:24:33.927496 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.927396 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975"} err="failed to get container status \"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975\": rpc error: code = NotFound desc = could not find container \"2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975\": container with ID starting with 2e1c2346d15eabf686945e668a14797d2eaa26dac634af3f9e88d89d56b66975 not found: ID does not exist" Oct 31 23:24:33.940058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.940026 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:24:33.943869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.943840 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-cvcfk"] Oct 31 23:24:33.967603 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.967552 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:24:33.967785 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:33.967627 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-7nxxv"] Oct 31 23:24:34.147489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.147453 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" path="/var/lib/kubelet/pods/0825b221-898d-4dcc-bdef-fa0f8673a2c8/volumes" Oct 31 23:24:34.148151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.148134 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" path="/var/lib/kubelet/pods/15558e00-bb89-4cac-9cd4-866a5f89d589/volumes" Oct 31 23:24:34.425738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.425711 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:24:34.485298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485260 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485334 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485568 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485498 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485629 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485609 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485637 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485755 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485683 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485755 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485720 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485865 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485843 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.485918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.485902 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\") pod \"821e4f82-82fa-47a5-b918-c2e782987f36\" (UID: \"821e4f82-82fa-47a5-b918-c2e782987f36\") " Oct 31 23:24:34.487133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.486684 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.487133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.486789 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.487133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.486869 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.487133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.487096 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.487133 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.487121 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.487510 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.487138 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:34.488060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.487974 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.488217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.488087 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:34.491951 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.491919 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.492114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.492033 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr" pod "821e4f82-82fa-47a5-b918-c2e782987f36" (UID: "821e4f82-82fa-47a5-b918-c2e782987f36"). InnerVolumeSpecName "kube-api-access-h47lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.588389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588349 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821e4f82-82fa-47a5-b918-c2e782987f36-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588387 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588402 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588415 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588429 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47lr\" (UniqueName: \"kubernetes.io/projected/821e4f82-82fa-47a5-b918-c2e782987f36-kube-api-access-h47lr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588443 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821e4f82-82fa-47a5-b918-c2e782987f36-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588457 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.588597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.588470 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821e4f82-82fa-47a5-b918-c2e782987f36-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.591371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.591344 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:24:34.619238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.618656 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:24:34.916131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.916092 7074 generic.go:356] "Generic (PLEG): container finished" podID="821e4f82-82fa-47a5-b918-c2e782987f36" containerID="b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e" exitCode=0 Oct 31 23:24:34.916328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.916143 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" Oct 31 23:24:34.916328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.916180 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" event={"ID":"821e4f82-82fa-47a5-b918-c2e782987f36","Type":"ContainerDied","Data":"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e"} Oct 31 23:24:34.916328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.916247 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg" event={"ID":"821e4f82-82fa-47a5-b918-c2e782987f36","Type":"ContainerDied","Data":"f7f2cfbc1f1a516438ffcfe2c6aef73ff8fea1df683183e69985f8f220336b46"} Oct 31 23:24:34.916328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.916282 7074 scope.go:122] "RemoveContainer" containerID="b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e" Oct 31 23:24:34.930444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.930420 7074 scope.go:122] "RemoveContainer" containerID="b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e" Oct 31 23:24:34.930835 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:34.930808 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e\": container with ID starting with b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e not found: ID does not exist" containerID="b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e" Oct 31 23:24:34.930937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.930847 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e"} err="failed to get container status \"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e\": rpc error: code = NotFound desc = could not find container \"b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e\": container with ID starting with b503ad284cb2c93396af53bc748da34e276afd23db58a799212fc38cd49f294e not found: ID does not exist" Oct 31 23:24:34.943940 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.943894 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:24:34.947777 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:34.947750 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-bb2lg"] Oct 31 23:24:35.422525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.422487 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:24:35.422806 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.422749 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" podUID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" containerName="maas-api" containerID="cri-o://75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af" gracePeriod=30 Oct 31 23:24:35.781016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.780389 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:24:35.936087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.936041 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\") pod \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " Oct 31 23:24:35.936585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.936515 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\") pod \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " Oct 31 23:24:35.936725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.936596 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") pod \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " Oct 31 23:24:35.936725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.936634 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\" (UniqueName: \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\") pod \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\" (UID: \"89021da0-1ab9-4a7c-bda0-4b465e3b801b\") " Oct 31 23:24:35.937665 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.937632 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca" pod "89021da0-1ab9-4a7c-bda0-4b465e3b801b" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:35.938374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.938344 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89021da0-1ab9-4a7c-bda0-4b465e3b801b-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.958640 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.958439 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs" pod "89021da0-1ab9-4a7c-bda0-4b465e3b801b" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:35.959428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.959398 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls" pod "89021da0-1ab9-4a7c-bda0-4b465e3b801b" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:35.962487 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.962455 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh" pod "89021da0-1ab9-4a7c-bda0-4b465e3b801b" (UID: "89021da0-1ab9-4a7c-bda0-4b465e3b801b"). InnerVolumeSpecName "kube-api-access-tg7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:35.977720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.975472 7074 generic.go:356] "Generic (PLEG): container finished" podID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" containerID="75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af" exitCode=0 Oct 31 23:24:35.977720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.975540 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" event={"ID":"89021da0-1ab9-4a7c-bda0-4b465e3b801b","Type":"ContainerDied","Data":"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af"} Oct 31 23:24:35.977720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.975605 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" event={"ID":"89021da0-1ab9-4a7c-bda0-4b465e3b801b","Type":"ContainerDied","Data":"d6923adc22a68b4fd02b04ce8d9bebad7f096c07059221856403d195a8d16609"} Oct 31 23:24:35.977720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.975635 7074 scope.go:122] "RemoveContainer" containerID="75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af" Oct 31 23:24:35.977720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.975725 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7" Oct 31 23:24:35.994840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.994789 7074 scope.go:122] "RemoveContainer" containerID="75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af" Oct 31 23:24:35.995227 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:35.995205 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af\": container with ID starting with 75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af not found: ID does not exist" containerID="75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af" Oct 31 23:24:35.995291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:35.995258 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af"} err="failed to get container status \"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af\": rpc error: code = NotFound desc = could not find container \"75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af\": container with ID starting with 75101799d5274da3f6d754d9dcbfa5167e5533e9ba31780bce6b3d9ec09831af not found: ID does not exist" Oct 31 23:24:36.009086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.009044 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:24:36.013115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.013086 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-270e8c-69d7699f8c-7z7c7"] Oct 31 23:24:36.040089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.040052 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:36.040089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.040088 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89021da0-1ab9-4a7c-bda0-4b465e3b801b-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:36.040401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.040104 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7kh\" (UniqueName: \"kubernetes.io/projected/89021da0-1ab9-4a7c-bda0-4b465e3b801b-kube-api-access-tg7kh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:36.145160 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.145128 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" path="/var/lib/kubelet/pods/821e4f82-82fa-47a5-b918-c2e782987f36/volumes" Oct 31 23:24:36.145674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:36.145654 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" path="/var/lib/kubelet/pods/89021da0-1ab9-4a7c-bda0-4b465e3b801b/volumes" Oct 31 23:24:37.375580 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.375540 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:24:37.376145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.375779 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" podUID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" containerName="maas-api" containerID="cri-o://db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8" gracePeriod=30 Oct 31 23:24:37.822666 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.822635 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:24:37.881693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.881653 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\") pod \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " Oct 31 23:24:37.881901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.881726 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\" (UniqueName: \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\") pod \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " Oct 31 23:24:37.881901 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.881801 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\") pod \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " Oct 31 23:24:37.882055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.881932 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\") pod \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\" (UID: \"c54681c2-f396-49eb-a8a2-5c2fd00f3cac\") " Oct 31 23:24:37.882203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.882171 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca" pod "c54681c2-f396-49eb-a8a2-5c2fd00f3cac" (UID: "c54681c2-f396-49eb-a8a2-5c2fd00f3cac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:37.882745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.882720 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:37.884259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.884232 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls" pod "c54681c2-f396-49eb-a8a2-5c2fd00f3cac" (UID: "c54681c2-f396-49eb-a8a2-5c2fd00f3cac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:37.884358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.884327 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs" pod "c54681c2-f396-49eb-a8a2-5c2fd00f3cac" (UID: "c54681c2-f396-49eb-a8a2-5c2fd00f3cac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:37.884500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.884471 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22" pod "c54681c2-f396-49eb-a8a2-5c2fd00f3cac" (UID: "c54681c2-f396-49eb-a8a2-5c2fd00f3cac"). InnerVolumeSpecName "kube-api-access-7mp22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:37.984608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.984573 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mp22\" (UniqueName: \"kubernetes.io/projected/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-kube-api-access-7mp22\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:37.984608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.984603 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:37.984608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.984613 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54681c2-f396-49eb-a8a2-5c2fd00f3cac-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:37.994119 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.994079 7074 generic.go:356] "Generic (PLEG): container finished" podID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" containerID="db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8" exitCode=0 Oct 31 23:24:37.994307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.994118 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" event={"ID":"c54681c2-f396-49eb-a8a2-5c2fd00f3cac","Type":"ContainerDied","Data":"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8"} Oct 31 23:24:37.994307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.994200 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" event={"ID":"c54681c2-f396-49eb-a8a2-5c2fd00f3cac","Type":"ContainerDied","Data":"f5f29e279d4aacd6922c0b93ba2e68fce68f76f7740a12559b9a4cd52ed44cd2"} Oct 31 23:24:37.994307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.994134 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w" Oct 31 23:24:37.994307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:37.994222 7074 scope.go:122] "RemoveContainer" containerID="db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8" Oct 31 23:24:38.010135 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:38.010110 7074 scope.go:122] "RemoveContainer" containerID="db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8" Oct 31 23:24:38.010604 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:38.010575 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8\": container with ID starting with db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8 not found: ID does not exist" containerID="db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8" Oct 31 23:24:38.010709 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:38.010614 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8"} err="failed to get container status \"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8\": rpc error: code = NotFound desc = could not find container \"db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8\": container with ID starting with db6ab0db0ddec68e51fed0f70ea17c02a8c6f6d42c0ce04dbe265a21a8a07ff8 not found: ID does not exist" Oct 31 23:24:38.027501 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:38.027466 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:24:38.030817 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:38.030787 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf279c-789694d8f7-8hd9w"] Oct 31 23:24:38.145129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:38.145041 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" path="/var/lib/kubelet/pods/c54681c2-f396-49eb-a8a2-5c2fd00f3cac/volumes" Oct 31 23:24:40.783961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:40.783928 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:24:40.799257 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:40.799226 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:24:41.779845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:41.779811 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:24:41.780137 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:41.780096 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" podUID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" containerName="maas-api" containerID="cri-o://1eb272bf75b76fc4be0a08c4b72bfc19bd1cf9d3c4251c314ca1445f25705b24" gracePeriod=30 Oct 31 23:24:42.079522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.079212 7074 generic.go:356] "Generic (PLEG): container finished" podID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" containerID="1eb272bf75b76fc4be0a08c4b72bfc19bd1cf9d3c4251c314ca1445f25705b24" exitCode=0 Oct 31 23:24:42.079522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.079358 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" event={"ID":"374a01e6-c99b-48f0-8fcc-466bf1ea96cb","Type":"ContainerDied","Data":"1eb272bf75b76fc4be0a08c4b72bfc19bd1cf9d3c4251c314ca1445f25705b24"} Oct 31 23:24:42.181888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.181860 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:24:42.302132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.302088 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\" (UniqueName: \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\") pod \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " Oct 31 23:24:42.302336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.302194 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\") pod \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " Oct 31 23:24:42.302336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.302240 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\") pod \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " Oct 31 23:24:42.302460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.302391 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\") pod \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\" (UID: \"374a01e6-c99b-48f0-8fcc-466bf1ea96cb\") " Oct 31 23:24:42.302794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.302768 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca" pod "374a01e6-c99b-48f0-8fcc-466bf1ea96cb" (UID: "374a01e6-c99b-48f0-8fcc-466bf1ea96cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:42.303338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.303312 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.305519 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.305471 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs" pod "374a01e6-c99b-48f0-8fcc-466bf1ea96cb" (UID: "374a01e6-c99b-48f0-8fcc-466bf1ea96cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:42.306388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.306361 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls" pod "374a01e6-c99b-48f0-8fcc-466bf1ea96cb" (UID: "374a01e6-c99b-48f0-8fcc-466bf1ea96cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:42.308582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.308553 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj" pod "374a01e6-c99b-48f0-8fcc-466bf1ea96cb" (UID: "374a01e6-c99b-48f0-8fcc-466bf1ea96cb"). InnerVolumeSpecName "kube-api-access-wjhmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:42.415005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.414888 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.415301 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.415259 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhmj\" (UniqueName: \"kubernetes.io/projected/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-kube-api-access-wjhmj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.415457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:42.415425 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/374a01e6-c99b-48f0-8fcc-466bf1ea96cb-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:43.090899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.090859 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" event={"ID":"374a01e6-c99b-48f0-8fcc-466bf1ea96cb","Type":"ContainerDied","Data":"78b1a9eb784f889fd0bb51dfa795193ccadfd4dd8de542487ad34b879bc825f5"} Oct 31 23:24:43.091483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.090967 7074 scope.go:122] "RemoveContainer" containerID="1eb272bf75b76fc4be0a08c4b72bfc19bd1cf9d3c4251c314ca1445f25705b24" Oct 31 23:24:43.091483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.091168 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq" Oct 31 23:24:43.118832 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.118797 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:24:43.123430 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.123402 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7c4c91-6767866665-zrhrq"] Oct 31 23:24:43.519506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.519470 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:24:43.519808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.519739 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" containerID="cri-o://f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036" gracePeriod=30 Oct 31 23:24:43.530062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.526726 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:24:43.530062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.527039 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" containerID="cri-o://088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad" gracePeriod=30 Oct 31 23:24:43.541466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.540537 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:24:43.541466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:43.540798 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" containerID="cri-o://0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1" gracePeriod=30 Oct 31 23:24:44.146644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:44.146605 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" path="/var/lib/kubelet/pods/374a01e6-c99b-48f0-8fcc-466bf1ea96cb/volumes" Oct 31 23:24:44.892896 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:44.892784 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:24:45.784818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:45.784758 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" containerID="cri-o://6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc" gracePeriod=25 Oct 31 23:24:45.804481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:45.800283 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" containerID="cri-o://e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d" gracePeriod=25 Oct 31 23:24:46.583420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:46.583378 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:46.583420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:46.583397 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:48.855037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.854631 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:24:48.923434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.923390 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:24:48.969930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.969861 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970097 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970155 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970186 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970220 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970244 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970264 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.970295 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970304 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.971110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.970337 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\") pod \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\" (UID: \"db4f68aa-fee5-43a3-b3b0-c768f0a3e123\") " Oct 31 23:24:48.972282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.972251 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.972576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.972551 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:48.977833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.977398 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.978011 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.977949 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:24:48.984060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.980776 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.984060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.981799 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.984060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.982219 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.984448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.984187 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:48.984448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.984203 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:48.993209 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:48.992969 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl" pod "db4f68aa-fee5-43a3-b3b0-c768f0a3e123" (UID: "db4f68aa-fee5-43a3-b3b0-c768f0a3e123"). InnerVolumeSpecName "kube-api-access-5tfgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072539 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072659 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072701 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072845 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072934 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.072982 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073033 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073060 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073112 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073166 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073202 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073273 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073320 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073364 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073430 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073464 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073551 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\") pod \"d2f5523b-cb46-4d56-a70d-c8460db757d9\" (UID: \"d2f5523b-cb46-4d56-a70d-c8460db757d9\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.073636 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\") pod \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\" (UID: \"f8285b23-0cb3-409c-8bcd-e933d1e359e1\") " Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074310 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074325 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074337 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074346 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfgl\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-kube-api-access-5tfgl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074357 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074365 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074375 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074385 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.074637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.074393 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4f68aa-fee5-43a3-b3b0-c768f0a3e123-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.076780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.076355 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.079266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.078984 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "kube-api-access-ng9lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.079608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.079406 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.079608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.079426 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.079946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.079806 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.079946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.079836 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.081800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.081109 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.081800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.081475 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.081800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.081515 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.081800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.081734 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.084098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.083841 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.084098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.083885 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.085090 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.085053 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "kube-api-access-kcvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.085327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.085267 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.085517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.085491 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.086303 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.086283 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo" pod "f8285b23-0cb3-409c-8bcd-e933d1e359e1" (UID: "f8285b23-0cb3-409c-8bcd-e933d1e359e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:49.087170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.087149 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:49.087374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.087341 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token" pod "d2f5523b-cb46-4d56-a70d-c8460db757d9" (UID: "d2f5523b-cb46-4d56-a70d-c8460db757d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.175842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175796 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.175842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175830 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvpc\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-kube-api-access-kcvpc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.175842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175840 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.175842 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175849 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175859 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175867 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175878 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175886 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175894 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175902 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175909 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175917 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175925 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8285b23-0cb3-409c-8bcd-e933d1e359e1-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175934 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f5523b-cb46-4d56-a70d-c8460db757d9-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175942 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175952 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9lz\" (UniqueName: \"kubernetes.io/projected/f8285b23-0cb3-409c-8bcd-e933d1e359e1-kube-api-access-ng9lz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175960 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f5523b-cb46-4d56-a70d-c8460db757d9-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.176247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.175970 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8285b23-0cb3-409c-8bcd-e933d1e359e1-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.198621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.198582 7074 generic.go:356] "Generic (PLEG): container finished" podID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerID="088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad" exitCode=0 Oct 31 23:24:49.198873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.198777 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" event={"ID":"d2f5523b-cb46-4d56-a70d-c8460db757d9","Type":"ContainerDied","Data":"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad"} Oct 31 23:24:49.198873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.198818 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" event={"ID":"d2f5523b-cb46-4d56-a70d-c8460db757d9","Type":"ContainerDied","Data":"2d59b44d15da00b61e9f938b128cb72b7d69ac76c33b340fed9d16aca850876f"} Oct 31 23:24:49.198873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.198838 7074 scope.go:122] "RemoveContainer" containerID="088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad" Oct 31 23:24:49.199101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.198953 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd" Oct 31 23:24:49.201200 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.201099 7074 generic.go:356] "Generic (PLEG): container finished" podID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerID="0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1" exitCode=0 Oct 31 23:24:49.201315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.201252 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" event={"ID":"f8285b23-0cb3-409c-8bcd-e933d1e359e1","Type":"ContainerDied","Data":"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1"} Oct 31 23:24:49.201315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.201308 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" event={"ID":"f8285b23-0cb3-409c-8bcd-e933d1e359e1","Type":"ContainerDied","Data":"84f78492d845b9c3c8cb10932ce39d51937ef00477ad6a8fa9b19778a0244f44"} Oct 31 23:24:49.201433 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.201387 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886" Oct 31 23:24:49.211823 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.211793 7074 generic.go:356] "Generic (PLEG): container finished" podID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerID="f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036" exitCode=0 Oct 31 23:24:49.212813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.211846 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" Oct 31 23:24:49.212813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.211873 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" event={"ID":"db4f68aa-fee5-43a3-b3b0-c768f0a3e123","Type":"ContainerDied","Data":"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036"} Oct 31 23:24:49.212813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.211932 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt" event={"ID":"db4f68aa-fee5-43a3-b3b0-c768f0a3e123","Type":"ContainerDied","Data":"60e3dd84c20b69102910ffcbe78acc030a53beee9c5795522c1d6e7cc0322842"} Oct 31 23:24:49.214938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.214918 7074 scope.go:122] "RemoveContainer" containerID="088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad" Oct 31 23:24:49.215342 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:49.215310 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad\": container with ID starting with 088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad not found: ID does not exist" containerID="088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad" Oct 31 23:24:49.215444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.215353 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad"} err="failed to get container status \"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad\": rpc error: code = NotFound desc = could not find container \"088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad\": container with ID starting with 088cac616117e56169413418e01678b742c13bda18914407325810c0fb602dad not found: ID does not exist" Oct 31 23:24:49.215444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.215379 7074 scope.go:122] "RemoveContainer" containerID="0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1" Oct 31 23:24:49.230965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.230928 7074 scope.go:122] "RemoveContainer" containerID="0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1" Oct 31 23:24:49.231385 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:49.231350 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1\": container with ID starting with 0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1 not found: ID does not exist" containerID="0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1" Oct 31 23:24:49.231462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.231401 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1"} err="failed to get container status \"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1\": rpc error: code = NotFound desc = could not find container \"0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1\": container with ID starting with 0723cdf5762f3408dafffb17cdbc192fbbc1643499077d3ba350f00b3b7249c1 not found: ID does not exist" Oct 31 23:24:49.231462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.231428 7074 scope.go:122] "RemoveContainer" containerID="f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036" Oct 31 23:24:49.242527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.242493 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:24:49.246025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.245984 7074 scope.go:122] "RemoveContainer" containerID="f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036" Oct 31 23:24:49.246365 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:49.246346 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036\": container with ID starting with f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036 not found: ID does not exist" containerID="f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036" Oct 31 23:24:49.246440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.246374 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036"} err="failed to get container status \"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036\": rpc error: code = NotFound desc = could not find container \"f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036\": container with ID starting with f631667bd1b4ef462bdf33ea844057055dbffe964be6a92c47f904ccc67bf036 not found: ID does not exist" Oct 31 23:24:49.246549 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.246533 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-plckd"] Oct 31 23:24:49.262214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.262180 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:24:49.271759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.271724 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-vd886"] Oct 31 23:24:49.291030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.290968 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:24:49.294970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.294937 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-p65pt"] Oct 31 23:24:49.614826 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.614785 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" containerID="cri-o://f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab" gracePeriod=45 Oct 31 23:24:49.645959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.645888 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" containerID="cri-o://307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b" gracePeriod=45 Oct 31 23:24:49.660875 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.660831 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Oct 31 23:24:49.965724 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:49.965640 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:24:50.005930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.005897 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:24:50.092024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.091920 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.091964 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092078 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092155 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092216 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.092336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092262 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\" (UniqueName: \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092311 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\") pod \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\" (UID: \"db1b6bef-79f3-4d65-98a7-e8932e6afdc1\") " Oct 31 23:24:50.092495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092458 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.092798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092759 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.092969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092948 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.093061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.092974 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.093061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.093003 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.094576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.094550 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.094706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.094689 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "kube-api-access-jvktl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:50.094845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.094830 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs" pod "db1b6bef-79f3-4d65-98a7-e8932e6afdc1" (UID: "db1b6bef-79f3-4d65-98a7-e8932e6afdc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:50.150825 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.150793 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" path="/var/lib/kubelet/pods/d2f5523b-cb46-4d56-a70d-c8460db757d9/volumes" Oct 31 23:24:50.151355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.151340 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" path="/var/lib/kubelet/pods/db4f68aa-fee5-43a3-b3b0-c768f0a3e123/volumes" Oct 31 23:24:50.151849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.151833 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" path="/var/lib/kubelet/pods/f8285b23-0cb3-409c-8bcd-e933d1e359e1/volumes" Oct 31 23:24:50.193979 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.193944 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\" (UniqueName: \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194127 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194240 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194223 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\" (UniqueName: \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194270 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194341 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194377 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\") pod \"dea39a53-3824-4c95-bb01-edf9a0383e67\" (UID: \"dea39a53-3824-4c95-bb01-edf9a0383e67\") " Oct 31 23:24:50.194794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194777 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.194873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194796 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.194873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.194807 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvktl\" (UniqueName: \"kubernetes.io/projected/db1b6bef-79f3-4d65-98a7-e8932e6afdc1-kube-api-access-jvktl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.195054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.195033 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.195547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.195519 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.195786 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.195766 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.197169 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.197139 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:50.197272 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.197251 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.197391 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.197369 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7" pod "dea39a53-3824-4c95-bb01-edf9a0383e67" (UID: "dea39a53-3824-4c95-bb01-edf9a0383e67"). InnerVolumeSpecName "kube-api-access-pfdx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:50.221984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.221948 7074 generic.go:356] "Generic (PLEG): container finished" podID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerID="f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab" exitCode=0 Oct 31 23:24:50.222194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.222030 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" Oct 31 23:24:50.222194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.222027 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" event={"ID":"db1b6bef-79f3-4d65-98a7-e8932e6afdc1","Type":"ContainerDied","Data":"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab"} Oct 31 23:24:50.222194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.222153 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc" event={"ID":"db1b6bef-79f3-4d65-98a7-e8932e6afdc1","Type":"ContainerDied","Data":"6e90274b234ed9fce7283ada9fc723e7c7a845e8d7ca086fc63813da839d865a"} Oct 31 23:24:50.222194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.222184 7074 scope.go:122] "RemoveContainer" containerID="f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab" Oct 31 23:24:50.226354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.226327 7074 generic.go:356] "Generic (PLEG): container finished" podID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerID="307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b" exitCode=0 Oct 31 23:24:50.226479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.226409 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" event={"ID":"dea39a53-3824-4c95-bb01-edf9a0383e67","Type":"ContainerDied","Data":"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b"} Oct 31 23:24:50.226479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.226450 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" event={"ID":"dea39a53-3824-4c95-bb01-edf9a0383e67","Type":"ContainerDied","Data":"574f2d01a4bc7d9d1fb8254bae05aac4d238a4d953e441ac01f8b43098e89197"} Oct 31 23:24:50.226479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.226417 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7" Oct 31 23:24:50.234794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.234766 7074 scope.go:122] "RemoveContainer" containerID="f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab" Oct 31 23:24:50.235203 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:50.235179 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab\": container with ID starting with f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab not found: ID does not exist" containerID="f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab" Oct 31 23:24:50.235294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.235214 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab"} err="failed to get container status \"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab\": rpc error: code = NotFound desc = could not find container \"f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab\": container with ID starting with f40251a1e3b32391af62c3cca218e6ae085c4d26fe361f94bb451bae3788ecab not found: ID does not exist" Oct 31 23:24:50.235294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.235237 7074 scope.go:122] "RemoveContainer" containerID="307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b" Oct 31 23:24:50.246872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.246721 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:24:50.247400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.247271 7074 scope.go:122] "RemoveContainer" containerID="307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b" Oct 31 23:24:50.247688 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:50.247668 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b\": container with ID starting with 307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b not found: ID does not exist" containerID="307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b" Oct 31 23:24:50.247765 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.247696 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b"} err="failed to get container status \"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b\": rpc error: code = NotFound desc = could not find container \"307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b\": container with ID starting with 307a0370fb793720002146d657701775066cecbfea54c67c7045ec80348fce5b not found: ID does not exist" Oct 31 23:24:50.251085 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.251057 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/distinct-w6-7d7eed-kserve-5d8bb5d5fc-qnnnc"] Oct 31 23:24:50.266280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.266234 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:24:50.270271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.270239 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/simulator-w6-7d7eed-kserve-649b9878c5-sd8k7"] Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.305948 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.306019 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.306036 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea39a53-3824-4c95-bb01-edf9a0383e67-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.306049 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.306065 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdx7\" (UniqueName: \"kubernetes.io/projected/dea39a53-3824-4c95-bb01-edf9a0383e67-kube-api-access-pfdx7\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.310388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:50.306079 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea39a53-3824-4c95-bb01-edf9a0383e67-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:52.142979 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:52.142949 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" path="/var/lib/kubelet/pods/db1b6bef-79f3-4d65-98a7-e8932e6afdc1/volumes" Oct 31 23:24:52.143616 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:52.143596 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" path="/var/lib/kubelet/pods/dea39a53-3824-4c95-bb01-edf9a0383e67/volumes" Oct 31 23:24:54.295095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:54.295058 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:24:54.312772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:54.312718 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:24:56.210086 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.210051 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:24:56.512979 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.512929 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:24:56.513354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.513260 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" containerID="cri-o://2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4" gracePeriod=30 Oct 31 23:24:56.524559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.518388 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:24:56.524559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.518734 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" containerID="cri-o://f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5" gracePeriod=30 Oct 31 23:24:56.633507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.633027 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:56.640340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.636919 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:56.678447 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.678071 7074 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:56.678447 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:56.678151 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:57.182723 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.182687 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:24:57.182968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.182904 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" podUID="e6d42afb-923e-490b-b388-d2a935f8f12a" containerName="maas-api" containerID="cri-o://89f6145d8d7faa9f5bf602daa844fcddfaaa2810f0a82f8d3427e84011bf62e4" gracePeriod=30 Oct 31 23:24:57.329923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.329894 7074 generic.go:356] "Generic (PLEG): container finished" podID="e6d42afb-923e-490b-b388-d2a935f8f12a" containerID="89f6145d8d7faa9f5bf602daa844fcddfaaa2810f0a82f8d3427e84011bf62e4" exitCode=0 Oct 31 23:24:57.330311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.329973 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" event={"ID":"e6d42afb-923e-490b-b388-d2a935f8f12a","Type":"ContainerDied","Data":"89f6145d8d7faa9f5bf602daa844fcddfaaa2810f0a82f8d3427e84011bf62e4"} Oct 31 23:24:57.741871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.741843 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.761455 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\") pod \"e6d42afb-923e-490b-b388-d2a935f8f12a\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.761519 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\") pod \"e6d42afb-923e-490b-b388-d2a935f8f12a\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.761635 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\" (UniqueName: \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\") pod \"e6d42afb-923e-490b-b388-d2a935f8f12a\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.761738 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\") pod \"e6d42afb-923e-490b-b388-d2a935f8f12a\" (UID: \"e6d42afb-923e-490b-b388-d2a935f8f12a\") " Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.763866 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca" pod "e6d42afb-923e-490b-b388-d2a935f8f12a" (UID: "e6d42afb-923e-490b-b388-d2a935f8f12a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:57.769613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.768838 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl" pod "e6d42afb-923e-490b-b388-d2a935f8f12a" (UID: "e6d42afb-923e-490b-b388-d2a935f8f12a"). InnerVolumeSpecName "kube-api-access-zprhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:57.789128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.779976 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls" pod "e6d42afb-923e-490b-b388-d2a935f8f12a" (UID: "e6d42afb-923e-490b-b388-d2a935f8f12a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:57.816253 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.816215 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs" pod "e6d42afb-923e-490b-b388-d2a935f8f12a" (UID: "e6d42afb-923e-490b-b388-d2a935f8f12a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:57.864046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.863965 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:57.864046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.864042 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d42afb-923e-490b-b388-d2a935f8f12a-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:57.864267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.864058 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d42afb-923e-490b-b388-d2a935f8f12a-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:57.864267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:57.864072 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zprhl\" (UniqueName: \"kubernetes.io/projected/e6d42afb-923e-490b-b388-d2a935f8f12a-kube-api-access-zprhl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:58.339320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.339288 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" Oct 31 23:24:58.339814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.339319 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6" event={"ID":"e6d42afb-923e-490b-b388-d2a935f8f12a","Type":"ContainerDied","Data":"a919986a1b5fd4ed6eaf174b14c704ea7811a61e98e78b4494f3ccb5ea293fa9"} Oct 31 23:24:58.339814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.339384 7074 scope.go:122] "RemoveContainer" containerID="89f6145d8d7faa9f5bf602daa844fcddfaaa2810f0a82f8d3427e84011bf62e4" Oct 31 23:24:58.362956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.362919 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:24:58.366633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.366606 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c4d32e-b99549b5c-pgbt6"] Oct 31 23:24:58.440170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.440086 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:24:58.440331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.440309 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" podUID="6db814a8-8d7b-40cb-b938-788f64abf26a" containerName="maas-api" containerID="cri-o://368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7" gracePeriod=30 Oct 31 23:24:58.782061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.782035 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:24:58.878383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.878349 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\") pod \"6db814a8-8d7b-40cb-b938-788f64abf26a\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " Oct 31 23:24:58.878581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.878442 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\") pod \"6db814a8-8d7b-40cb-b938-788f64abf26a\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " Oct 31 23:24:58.878581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.878503 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\" (UniqueName: \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\") pod \"6db814a8-8d7b-40cb-b938-788f64abf26a\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " Oct 31 23:24:58.878581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.878528 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\") pod \"6db814a8-8d7b-40cb-b938-788f64abf26a\" (UID: \"6db814a8-8d7b-40cb-b938-788f64abf26a\") " Oct 31 23:24:58.878895 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.878866 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca" pod "6db814a8-8d7b-40cb-b938-788f64abf26a" (UID: "6db814a8-8d7b-40cb-b938-788f64abf26a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:58.879262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.879240 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db814a8-8d7b-40cb-b938-788f64abf26a-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:58.880753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.880716 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj" pod "6db814a8-8d7b-40cb-b938-788f64abf26a" (UID: "6db814a8-8d7b-40cb-b938-788f64abf26a"). InnerVolumeSpecName "kube-api-access-zcfpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:58.880837 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.880776 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs" pod "6db814a8-8d7b-40cb-b938-788f64abf26a" (UID: "6db814a8-8d7b-40cb-b938-788f64abf26a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:58.880934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.880913 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls" pod "6db814a8-8d7b-40cb-b938-788f64abf26a" (UID: "6db814a8-8d7b-40cb-b938-788f64abf26a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:58.982012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.980739 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:58.982012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.980784 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfpj\" (UniqueName: \"kubernetes.io/projected/6db814a8-8d7b-40cb-b938-788f64abf26a-kube-api-access-zcfpj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:58.982012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:58.980800 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db814a8-8d7b-40cb-b938-788f64abf26a-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:59.295936 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.295890 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" containerID="cri-o://2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc" gracePeriod=25 Oct 31 23:24:59.314291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.313967 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" containerID="cri-o://58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b" gracePeriod=25 Oct 31 23:24:59.352638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.352602 7074 generic.go:356] "Generic (PLEG): container finished" podID="6db814a8-8d7b-40cb-b938-788f64abf26a" containerID="368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7" exitCode=0 Oct 31 23:24:59.353144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.352762 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" event={"ID":"6db814a8-8d7b-40cb-b938-788f64abf26a","Type":"ContainerDied","Data":"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7"} Oct 31 23:24:59.353144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.352806 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" event={"ID":"6db814a8-8d7b-40cb-b938-788f64abf26a","Type":"ContainerDied","Data":"acf8fb1973cc1d5ead63412dcb9ad89db5efc0fd6513d5453ef35966fcd9dd20"} Oct 31 23:24:59.353144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.352829 7074 scope.go:122] "RemoveContainer" containerID="368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7" Oct 31 23:24:59.353144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.352909 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v" Oct 31 23:24:59.377233 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.377116 7074 scope.go:122] "RemoveContainer" containerID="368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7" Oct 31 23:24:59.377559 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:24:59.377538 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7\": container with ID starting with 368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7 not found: ID does not exist" containerID="368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7" Oct 31 23:24:59.377675 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.377657 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7"} err="failed to get container status \"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7\": rpc error: code = NotFound desc = could not find container \"368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7\": container with ID starting with 368339beebabb7a25f72758637ca2c69dd2a1d14da2ff87a372c74756c2a94e7 not found: ID does not exist" Oct 31 23:24:59.394551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.394521 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:24:59.397769 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:24:59.397739 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-135248-f745d847d-fgg8v"] Oct 31 23:25:00.143745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:00.143714 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db814a8-8d7b-40cb-b938-788f64abf26a" path="/var/lib/kubelet/pods/6db814a8-8d7b-40cb-b938-788f64abf26a/volumes" Oct 31 23:25:00.144235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:00.144221 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d42afb-923e-490b-b388-d2a935f8f12a" path="/var/lib/kubelet/pods/e6d42afb-923e-490b-b388-d2a935f8f12a/volumes" Oct 31 23:25:01.443980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:01.443940 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:25:01.461850 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:01.461813 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:25:01.689911 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:01.689779 7074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384aa41f_ca9b_4287_b210_856ebe1eab8a.slice/crio-f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5.scope\": RecentStats: unable to find data in memory cache]" Oct 31 23:25:01.993274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:01.993196 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:25:02.040737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.040689 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.040737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.040737 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.040788 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.040880 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.041040 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.041112 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.041234 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.041268 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.041304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.041297 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\") pod \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\" (UID: \"31668b3b-cfeb-4832-a16d-3ac77c8d6694\") " Oct 31 23:25:02.042380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042072 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.042380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042142 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.042380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042211 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.042380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042378 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:02.042783 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042761 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.043028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.042970 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:25:02.048660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.044230 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "kube-api-access-tbwd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.048660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.046457 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.048660 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.048619 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.051569 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.050328 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo" pod "31668b3b-cfeb-4832-a16d-3ac77c8d6694" (UID: "31668b3b-cfeb-4832-a16d-3ac77c8d6694"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:02.142773 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.142739 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.142949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.142800 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.142949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.142898 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.142949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.142926 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.142955 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143108 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143140 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143244 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143280 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\") pod \"384aa41f-ca9b-4287-b210-856ebe1eab8a\" (UID: \"384aa41f-ca9b-4287-b210-856ebe1eab8a\") " Oct 31 23:25:02.143442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143340 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:02.143497 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143459 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.143921 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144025 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144048 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144067 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144082 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384aa41f-ca9b-4287-b210-856ebe1eab8a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144096 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144110 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwd2\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-kube-api-access-tbwd2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144126 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144140 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144154 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144168 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31668b3b-cfeb-4832-a16d-3ac77c8d6694-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144173 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.143984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.144184 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31668b3b-cfeb-4832-a16d-3ac77c8d6694-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.145235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.145206 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.145716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.145689 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.145828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.145729 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:02.145828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.145768 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:02.145828 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.145774 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn" pod "384aa41f-ca9b-4287-b210-856ebe1eab8a" (UID: "384aa41f-ca9b-4287-b210-856ebe1eab8a"). InnerVolumeSpecName "kube-api-access-jpdqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246162 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246197 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246208 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdqn\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-kube-api-access-jpdqn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246219 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246228 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246237 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.246252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.246246 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384aa41f-ca9b-4287-b210-856ebe1eab8a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.387332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.387294 7074 generic.go:356] "Generic (PLEG): container finished" podID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerID="2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4" exitCode=0 Oct 31 23:25:02.387490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.387347 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" Oct 31 23:25:02.387490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.387385 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" event={"ID":"31668b3b-cfeb-4832-a16d-3ac77c8d6694","Type":"ContainerDied","Data":"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4"} Oct 31 23:25:02.387490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.387440 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9" event={"ID":"31668b3b-cfeb-4832-a16d-3ac77c8d6694","Type":"ContainerDied","Data":"963d4375e171177c6d56de9fa837b341bb6ed547298348ad3a729199b46283e7"} Oct 31 23:25:02.387490 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.387462 7074 scope.go:122] "RemoveContainer" containerID="2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4" Oct 31 23:25:02.389198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.389168 7074 generic.go:356] "Generic (PLEG): container finished" podID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerID="f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5" exitCode=0 Oct 31 23:25:02.389294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.389216 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" Oct 31 23:25:02.389294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.389245 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" event={"ID":"384aa41f-ca9b-4287-b210-856ebe1eab8a","Type":"ContainerDied","Data":"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5"} Oct 31 23:25:02.389384 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.389293 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll" event={"ID":"384aa41f-ca9b-4287-b210-856ebe1eab8a","Type":"ContainerDied","Data":"d7be599194f0e2658859fe475026f8da6b6fa657ee5fd73d6b7d30a909c8dd3d"} Oct 31 23:25:02.398952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.398932 7074 scope.go:122] "RemoveContainer" containerID="2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4" Oct 31 23:25:02.399380 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:02.399352 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4\": container with ID starting with 2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4 not found: ID does not exist" containerID="2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4" Oct 31 23:25:02.399477 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.399391 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4"} err="failed to get container status \"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4\": rpc error: code = NotFound desc = could not find container \"2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4\": container with ID starting with 2284fafaa6a57316555eca50acaf40d9dd1932562dba3e243686d3e2995dd5c4 not found: ID does not exist" Oct 31 23:25:02.399477 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.399417 7074 scope.go:122] "RemoveContainer" containerID="f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5" Oct 31 23:25:02.409430 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.409400 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:25:02.410817 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.410781 7074 scope.go:122] "RemoveContainer" containerID="f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5" Oct 31 23:25:02.411165 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:02.411144 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5\": container with ID starting with f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5 not found: ID does not exist" containerID="f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5" Oct 31 23:25:02.411234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.411177 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5"} err="failed to get container status \"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5\": rpc error: code = NotFound desc = could not find container \"f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5\": container with ID starting with f90fda8683cabe5bde6588ab373a1e6a679589d4fdf60eee6cc9ec2885a2edd5 not found: ID does not exist" Oct 31 23:25:02.417543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.417518 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-8j5m9"] Oct 31 23:25:02.434620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.434586 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:25:02.438635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:02.438606 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-nftll"] Oct 31 23:25:04.145146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:04.145067 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" path="/var/lib/kubelet/pods/31668b3b-cfeb-4832-a16d-3ac77c8d6694/volumes" Oct 31 23:25:04.145700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:04.145683 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" path="/var/lib/kubelet/pods/384aa41f-ca9b-4287-b210-856ebe1eab8a/volumes" Oct 31 23:25:06.132637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.132565 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:25:06.161324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.161290 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:25:06.224429 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.224387 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\" (UniqueName: \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\") pod \"6c96086e-8783-457f-9aa5-dd76960b42b5\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " Oct 31 23:25:06.224634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.224517 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\" (UniqueName: \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\") pod \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " Oct 31 23:25:06.224634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.224583 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\") pod \"6c96086e-8783-457f-9aa5-dd76960b42b5\" (UID: \"6c96086e-8783-457f-9aa5-dd76960b42b5\") " Oct 31 23:25:06.224634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.224623 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\") pod \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\" (UID: \"46131fa4-7daa-4a14-bda7-dcc36fc6eedb\") " Oct 31 23:25:06.225029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.224999 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume" pod "6c96086e-8783-457f-9aa5-dd76960b42b5" (UID: "6c96086e-8783-457f-9aa5-dd76960b42b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:06.225144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.225125 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume" pod "46131fa4-7daa-4a14-bda7-dcc36fc6eedb" (UID: "46131fa4-7daa-4a14-bda7-dcc36fc6eedb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:06.225422 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.225396 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:06.225513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.225430 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c96086e-8783-457f-9aa5-dd76960b42b5-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:06.227617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.226671 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln" pod "46131fa4-7daa-4a14-bda7-dcc36fc6eedb" (UID: "46131fa4-7daa-4a14-bda7-dcc36fc6eedb"). InnerVolumeSpecName "kube-api-access-6xdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:06.227617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.227182 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl" pod "6c96086e-8783-457f-9aa5-dd76960b42b5" (UID: "6c96086e-8783-457f-9aa5-dd76960b42b5"). InnerVolumeSpecName "kube-api-access-c8fxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:06.326454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.326411 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8fxl\" (UniqueName: \"kubernetes.io/projected/6c96086e-8783-457f-9aa5-dd76960b42b5-kube-api-access-c8fxl\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:06.326454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.326447 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xdln\" (UniqueName: \"kubernetes.io/projected/46131fa4-7daa-4a14-bda7-dcc36fc6eedb-kube-api-access-6xdln\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:06.428733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.428631 7074 generic.go:356] "Generic (PLEG): container finished" podID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerID="6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc" exitCode=0 Oct 31 23:25:06.428733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.428696 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" Oct 31 23:25:06.428733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.428717 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" event={"ID":"6c96086e-8783-457f-9aa5-dd76960b42b5","Type":"ContainerDied","Data":"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc"} Oct 31 23:25:06.429054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.428778 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh" event={"ID":"6c96086e-8783-457f-9aa5-dd76960b42b5","Type":"ContainerDied","Data":"d0fdbe5f9f847c60d9816d33dbf80f93ccbd1049db2d3cccf1ee5a3be29ad446"} Oct 31 23:25:06.429054 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.428816 7074 scope.go:122] "RemoveContainer" containerID="6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc" Oct 31 23:25:06.430321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.430300 7074 generic.go:356] "Generic (PLEG): container finished" podID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerID="e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d" exitCode=0 Oct 31 23:25:06.430425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.430324 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" event={"ID":"46131fa4-7daa-4a14-bda7-dcc36fc6eedb","Type":"ContainerDied","Data":"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d"} Oct 31 23:25:06.430425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.430365 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" Oct 31 23:25:06.430425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.430372 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg" event={"ID":"46131fa4-7daa-4a14-bda7-dcc36fc6eedb","Type":"ContainerDied","Data":"d5c2162b399ec7ea367350c10d94d062d9bdf888384e85707ea3df2be2065279"} Oct 31 23:25:06.444412 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.444344 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" containerID="cri-o://e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1" gracePeriod=25 Oct 31 23:25:06.461236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.454945 7074 scope.go:122] "RemoveContainer" containerID="6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc" Oct 31 23:25:06.473222 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:06.472158 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc\": container with ID starting with 6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc not found: ID does not exist" containerID="6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc" Oct 31 23:25:06.473222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.472211 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc"} err="failed to get container status \"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc\": rpc error: code = NotFound desc = could not find container \"6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc\": container with ID starting with 6543c8f6ffcf424a3ba50eebab268e9439a7fbf4fa79b75aba40f24455b7eafc not found: ID does not exist" Oct 31 23:25:06.473222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.472240 7074 scope.go:122] "RemoveContainer" containerID="e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d" Oct 31 23:25:06.473222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.472914 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" containerID="cri-o://d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416" gracePeriod=25 Oct 31 23:25:06.501170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.501123 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:25:06.506451 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.505738 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-135248-66bf8fdd84-zd9vh"] Oct 31 23:25:06.541111 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.541076 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:25:06.543764 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.543690 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-135248-6466565b67-dmdrg"] Oct 31 23:25:06.546316 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.546253 7074 scope.go:122] "RemoveContainer" containerID="e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d" Oct 31 23:25:06.547263 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:06.547232 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d\": container with ID starting with e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d not found: ID does not exist" containerID="e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d" Oct 31 23:25:06.547398 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:06.547271 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d"} err="failed to get container status \"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d\": rpc error: code = NotFound desc = could not find container \"e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d\": container with ID starting with e59e1562ed57210253bc8bcf16dd8cb19f048732bdba01927683301861f1d12d not found: ID does not exist" Oct 31 23:25:07.565397 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:07.565361 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:25:07.565893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:07.565573 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" containerID="cri-o://0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4" gracePeriod=30 Oct 31 23:25:07.582638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:07.582602 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:25:07.582830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:07.582805 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" containerID="cri-o://aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4" gracePeriod=30 Oct 31 23:25:08.143566 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.143529 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" path="/var/lib/kubelet/pods/46131fa4-7daa-4a14-bda7-dcc36fc6eedb/volumes" Oct 31 23:25:08.144043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.143981 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" path="/var/lib/kubelet/pods/6c96086e-8783-457f-9aa5-dd76960b42b5/volumes" Oct 31 23:25:08.624134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.624091 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:08.624767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.624465 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:08.624851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.624805 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:08.625152 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:08.625126 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:09.245109 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:09.245072 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:25:09.245356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:09.245328 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" containerID="cri-o://abb45ee047311dbdf4c2fd5468750e93bff6789370bee851d91a9d88947b0fb6" gracePeriod=30 Oct 31 23:25:09.246928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:09.246902 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:25:09.247115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:09.247094 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" containerID="cri-o://ebdcda9c57981b856ba6fba3aee2402a9f21af8f47657b455d3220414aed01b1" gracePeriod=30 Oct 31 23:25:11.252274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.252226 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" containerID="cri-o://02aaa7609b34818325738145e7df305823b85cb31084a8426a74dc189d70a488" gracePeriod=45 Oct 31 23:25:11.490173 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.490129 7074 generic.go:356] "Generic (PLEG): container finished" podID="d64337a1-7f1d-4fce-ab33-79092a707849" containerID="02aaa7609b34818325738145e7df305823b85cb31084a8426a74dc189d70a488" exitCode=0 Oct 31 23:25:11.490355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.490196 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" event={"ID":"d64337a1-7f1d-4fce-ab33-79092a707849","Type":"ContainerDied","Data":"02aaa7609b34818325738145e7df305823b85cb31084a8426a74dc189d70a488"} Oct 31 23:25:11.553629 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.553581 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:25:11.609623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.609582 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.609824 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.609688 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.609938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.609913 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.610242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.609984 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.610242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.609917 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.610242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610025 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\" (UniqueName: \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.610242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610113 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\" (UniqueName: \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\") pod \"d64337a1-7f1d-4fce-ab33-79092a707849\" (UID: \"d64337a1-7f1d-4fce-ab33-79092a707849\") " Oct 31 23:25:11.610460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610272 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.610603 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610576 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.610897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610876 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.610945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610905 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.610945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.610922 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.612436 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.612409 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "kube-api-access-zsf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:11.612707 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.612683 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:11.616147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.616115 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm" pod "d64337a1-7f1d-4fce-ab33-79092a707849" (UID: "d64337a1-7f1d-4fce-ab33-79092a707849"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.678074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.678028 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:11.678239 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.678093 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:11.716053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.712982 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64337a1-7f1d-4fce-ab33-79092a707849-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.716053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.713043 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsf8q\" (UniqueName: \"kubernetes.io/projected/d64337a1-7f1d-4fce-ab33-79092a707849-kube-api-access-zsf8q\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.716053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.713059 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64337a1-7f1d-4fce-ab33-79092a707849-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.761899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.761808 7074 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:11.762121 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.761890 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:11.987471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.987432 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:25:11.987976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:11.987943 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" podUID="b8db220b-9983-4da1-b152-6bc107121c17" containerName="maas-api" containerID="cri-o://8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099" gracePeriod=30 Oct 31 23:25:12.374633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.374548 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:25:12.422815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.422775 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\") pod \"b8db220b-9983-4da1-b152-6bc107121c17\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " Oct 31 23:25:12.423153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.423121 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\") pod \"b8db220b-9983-4da1-b152-6bc107121c17\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " Oct 31 23:25:12.423289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.423277 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\") pod \"b8db220b-9983-4da1-b152-6bc107121c17\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " Oct 31 23:25:12.423360 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.423349 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\" (UniqueName: \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\") pod \"b8db220b-9983-4da1-b152-6bc107121c17\" (UID: \"b8db220b-9983-4da1-b152-6bc107121c17\") " Oct 31 23:25:12.423694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.423669 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca" pod "b8db220b-9983-4da1-b152-6bc107121c17" (UID: "b8db220b-9983-4da1-b152-6bc107121c17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:12.424036 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.424014 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8db220b-9983-4da1-b152-6bc107121c17-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:12.425776 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.425742 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt" pod "b8db220b-9983-4da1-b152-6bc107121c17" (UID: "b8db220b-9983-4da1-b152-6bc107121c17"). InnerVolumeSpecName "kube-api-access-fjxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:12.425776 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.425760 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs" pod "b8db220b-9983-4da1-b152-6bc107121c17" (UID: "b8db220b-9983-4da1-b152-6bc107121c17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:12.426050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.426025 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls" pod "b8db220b-9983-4da1-b152-6bc107121c17" (UID: "b8db220b-9983-4da1-b152-6bc107121c17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:12.498654 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.498622 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" Oct 31 23:25:12.498833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.498650 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv" event={"ID":"d64337a1-7f1d-4fce-ab33-79092a707849","Type":"ContainerDied","Data":"7951371c91e157cc8ab7075739b8fc8759e518c4f58d7aaf00d500f3b0f3974b"} Oct 31 23:25:12.498833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.498707 7074 scope.go:122] "RemoveContainer" containerID="02aaa7609b34818325738145e7df305823b85cb31084a8426a74dc189d70a488" Oct 31 23:25:12.500278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.500251 7074 generic.go:356] "Generic (PLEG): container finished" podID="b8db220b-9983-4da1-b152-6bc107121c17" containerID="8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099" exitCode=0 Oct 31 23:25:12.500396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.500291 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" Oct 31 23:25:12.500396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.500356 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" event={"ID":"b8db220b-9983-4da1-b152-6bc107121c17","Type":"ContainerDied","Data":"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099"} Oct 31 23:25:12.500506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.500403 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95" event={"ID":"b8db220b-9983-4da1-b152-6bc107121c17","Type":"ContainerDied","Data":"6336ecb0b7e7cb68e3f883a1c8a529cadffaeb3e5ac7504244b2ac2bce690823"} Oct 31 23:25:12.516481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.516460 7074 scope.go:122] "RemoveContainer" containerID="8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099" Oct 31 23:25:12.524159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.524129 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:25:12.525143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.525116 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxqt\" (UniqueName: \"kubernetes.io/projected/b8db220b-9983-4da1-b152-6bc107121c17-kube-api-access-fjxqt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:12.525242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.525147 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:12.525242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.525163 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8db220b-9983-4da1-b152-6bc107121c17-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:12.527166 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.527140 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/premium-w1-04c5c2-kserve-5657c99c46-4dcvv"] Oct 31 23:25:12.528548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.528530 7074 scope.go:122] "RemoveContainer" containerID="8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099" Oct 31 23:25:12.528876 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:12.528856 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099\": container with ID starting with 8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099 not found: ID does not exist" containerID="8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099" Oct 31 23:25:12.528923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.528884 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099"} err="failed to get container status \"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099\": rpc error: code = NotFound desc = could not find container \"8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099\": container with ID starting with 8b3769d9fcca1693a0926e1c547c54a3d4e612cfeff992676ec4503055028099 not found: ID does not exist" Oct 31 23:25:12.538783 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.538748 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:25:12.543360 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.543329 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04c5c2-857d954885-24g95"] Oct 31 23:25:12.817311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.817275 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:25:12.817559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.817532 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" containerID="cri-o://9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253" gracePeriod=30 Oct 31 23:25:12.868029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.868001 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:25:12.933104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.932914 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933041 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933135 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933174 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933205 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933251 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933330 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933423 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.933687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.933620 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\") pod \"a3248837-588d-4f99-8122-5517412e2120\" (UID: \"a3248837-588d-4f99-8122-5517412e2120\") " Oct 31 23:25:12.934236 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.934197 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:12.934536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.934509 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:12.934703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.934680 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:12.934950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.934928 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:12.935084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.935068 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:12.939016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.938952 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:12.939547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.939517 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:12.940346 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.940311 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:12.942497 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.942255 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4" pod "a3248837-588d-4f99-8122-5517412e2120" (UID: "a3248837-588d-4f99-8122-5517412e2120"). InnerVolumeSpecName "kube-api-access-vdgq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:12.978967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:12.978933 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035291 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035362 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035425 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035507 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035528 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035623 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035690 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035767 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.035836 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\") pod \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\" (UID: \"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85\") " Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036471 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036496 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036513 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036529 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3248837-588d-4f99-8122-5517412e2120-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036543 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdgq4\" (UniqueName: \"kubernetes.io/projected/a3248837-588d-4f99-8122-5517412e2120-kube-api-access-vdgq4\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036558 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036571 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036585 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3248837-588d-4f99-8122-5517412e2120-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.036791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.036594 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3248837-588d-4f99-8122-5517412e2120-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.037703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.037129 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.037703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.037464 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.038198 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.037893 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:13.038520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.038464 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.038951 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.038907 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.040932 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.040892 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.041249 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.041225 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:13.041364 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.041349 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.041553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.041519 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj" pod "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" (UID: "31c7f385-bc1e-43a1-8f3a-b0d7a44dba85"). InnerVolumeSpecName "kube-api-access-8v9dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.138283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138248 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138279 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138289 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138297 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138306 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138314 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138324 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9dj\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-kube-api-access-8v9dj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138332 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.138527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.138341 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.513344 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.513305 7074 generic.go:356] "Generic (PLEG): container finished" podID="a3248837-588d-4f99-8122-5517412e2120" containerID="0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4" exitCode=0 Oct 31 23:25:13.515267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.513858 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" event={"ID":"a3248837-588d-4f99-8122-5517412e2120","Type":"ContainerDied","Data":"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4"} Oct 31 23:25:13.515267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.514362 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" event={"ID":"a3248837-588d-4f99-8122-5517412e2120","Type":"ContainerDied","Data":"e2d6fc4ca4032d64845a5b10a20452cce16a4b0ba801b8a4abf0f6152bfa6fc7"} Oct 31 23:25:13.515267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.514412 7074 scope.go:122] "RemoveContainer" containerID="0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4" Oct 31 23:25:13.515267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.513929 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm" Oct 31 23:25:13.529462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.529427 7074 generic.go:356] "Generic (PLEG): container finished" podID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerID="aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4" exitCode=0 Oct 31 23:25:13.529669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.529471 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" event={"ID":"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85","Type":"ContainerDied","Data":"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4"} Oct 31 23:25:13.529669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.529557 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" event={"ID":"31c7f385-bc1e-43a1-8f3a-b0d7a44dba85","Type":"ContainerDied","Data":"cbc2cabb3bd722effeab4bee19d599ded0e8cdba7b7b128115954187fdf2b1b9"} Oct 31 23:25:13.529669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.529660 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8" Oct 31 23:25:13.552285 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.552241 7074 scope.go:122] "RemoveContainer" containerID="0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4" Oct 31 23:25:13.555408 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:13.555361 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4\": container with ID starting with 0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4 not found: ID does not exist" containerID="0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4" Oct 31 23:25:13.555576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.555419 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4"} err="failed to get container status \"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4\": rpc error: code = NotFound desc = could not find container \"0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4\": container with ID starting with 0ea858aa4ce219fb4be904a46fc3b188a23cf6748f185423b9be301c47c606e4 not found: ID does not exist" Oct 31 23:25:13.555576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.555448 7074 scope.go:122] "RemoveContainer" containerID="aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4" Oct 31 23:25:13.570692 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.570658 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:25:13.573316 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.573293 7074 scope.go:122] "RemoveContainer" containerID="aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4" Oct 31 23:25:13.573536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.573496 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-kqwhm"] Oct 31 23:25:13.573965 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:13.573899 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4\": container with ID starting with aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4 not found: ID does not exist" containerID="aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4" Oct 31 23:25:13.573965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.573940 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4"} err="failed to get container status \"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4\": rpc error: code = NotFound desc = could not find container \"aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4\": container with ID starting with aeee7bee301b6f2e3120a5f51132ca2ebc246ec4231bf88ae127d8741b5741c4 not found: ID does not exist" Oct 31 23:25:13.583161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.583131 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:25:13.586883 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:13.586845 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-6cjc8"] Oct 31 23:25:14.149145 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.149106 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" path="/var/lib/kubelet/pods/31c7f385-bc1e-43a1-8f3a-b0d7a44dba85/volumes" Oct 31 23:25:14.149682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.149664 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3248837-588d-4f99-8122-5517412e2120" path="/var/lib/kubelet/pods/a3248837-588d-4f99-8122-5517412e2120/volumes" Oct 31 23:25:14.150128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.150112 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8db220b-9983-4da1-b152-6bc107121c17" path="/var/lib/kubelet/pods/b8db220b-9983-4da1-b152-6bc107121c17/volumes" Oct 31 23:25:14.151746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.151248 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" path="/var/lib/kubelet/pods/d64337a1-7f1d-4fce-ab33-79092a707849/volumes" Oct 31 23:25:14.752927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.752563 7074 generic.go:356] "Generic (PLEG): container finished" podID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerID="abb45ee047311dbdf4c2fd5468750e93bff6789370bee851d91a9d88947b0fb6" exitCode=0 Oct 31 23:25:14.752927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.752842 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" event={"ID":"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9","Type":"ContainerDied","Data":"abb45ee047311dbdf4c2fd5468750e93bff6789370bee851d91a9d88947b0fb6"} Oct 31 23:25:14.755939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.755746 7074 generic.go:356] "Generic (PLEG): container finished" podID="c139df47-d3de-4818-805f-05810cdccb12" containerID="ebdcda9c57981b856ba6fba3aee2402a9f21af8f47657b455d3220414aed01b1" exitCode=0 Oct 31 23:25:14.755939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:14.755841 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" event={"ID":"c139df47-d3de-4818-805f-05810cdccb12","Type":"ContainerDied","Data":"ebdcda9c57981b856ba6fba3aee2402a9f21af8f47657b455d3220414aed01b1"} Oct 31 23:25:15.327255 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:15.293225 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:25:15.374137 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.374050 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:25:15.503029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.499264 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:25:15.539023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.531676 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.543448 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.543415 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.572235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.572196 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.572808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.572785 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.573477 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.573456 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.629250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.626954 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.629250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.627048 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.629250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.627099 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.629250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.627186 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\") pod \"c139df47-d3de-4818-805f-05810cdccb12\" (UID: \"c139df47-d3de-4818-805f-05810cdccb12\") " Oct 31 23:25:15.673531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.543225 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:15.673531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.572700 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.673531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.584099 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.673531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.673225 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.719733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.707341 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734120 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734174 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734218 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734252 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734292 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734337 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734385 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734521 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.734582 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\") pod \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\" (UID: \"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9\") " Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.735215 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c139df47-d3de-4818-805f-05810cdccb12-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.735243 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.735261 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.735274 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.737020 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.735287 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.762052 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.741335 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "kube-api-access-wxrxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.773018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.768593 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:15.773018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.768889 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.773018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.769109 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.803142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.784268 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.803142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.802473 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" event={"ID":"c139df47-d3de-4818-805f-05810cdccb12","Type":"ContainerDied","Data":"a1d066c669e44a25307e3cba470ac8d03957018420f9271174ffa16681562505"} Oct 31 23:25:15.803142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.802588 7074 scope.go:122] "RemoveContainer" containerID="ebdcda9c57981b856ba6fba3aee2402a9f21af8f47657b455d3220414aed01b1" Oct 31 23:25:15.803142 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.802762 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f" Oct 31 23:25:15.806414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.806111 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.807524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.807232 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" event={"ID":"e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9","Type":"ContainerDied","Data":"af5931a38b8cb72f95d67dd5f2e7ad2cf0f5a2908b6cb311fde9eebdcd5aa37a"} Oct 31 23:25:15.807524 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.807353 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt" Oct 31 23:25:15.815694 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.815644 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "kube-api-access-8r8xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.817526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.817480 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.817866 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.817832 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.818419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.818374 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:15.819456 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.819419 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token" pod "c139df47-d3de-4818-805f-05810cdccb12" (UID: "c139df47-d3de-4818-805f-05810cdccb12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.820784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.820741 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837192 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c139df47-d3de-4818-805f-05810cdccb12-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837234 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c139df47-d3de-4818-805f-05810cdccb12-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837249 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837263 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837276 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837290 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837303 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8xr\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-kube-api-access-8r8xr\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837319 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837333 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837348 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837361 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrxf\" (UniqueName: \"kubernetes.io/projected/c139df47-d3de-4818-805f-05810cdccb12-kube-api-access-wxrxf\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.837442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.837378 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.844589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.844521 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy" pod "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" (UID: "e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.877897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.877179 7074 scope.go:122] "RemoveContainer" containerID="abb45ee047311dbdf4c2fd5468750e93bff6789370bee851d91a9d88947b0fb6" Oct 31 23:25:15.941483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:15.941149 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:16.147045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:16.146919 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:25:16.149780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:16.149743 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-j98wt"] Oct 31 23:25:16.162804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:16.162655 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:25:16.167485 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:16.167453 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-42t5f"] Oct 31 23:25:17.381102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.381059 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:25:17.381647 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.381288 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" containerID="cri-o://956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" gracePeriod=30 Oct 31 23:25:17.384168 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.384139 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:25:17.384370 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.384344 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" containerID="cri-o://e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" gracePeriod=30 Oct 31 23:25:17.606063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.606028 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:25:17.606509 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.606431 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" podUID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" containerName="maas-api" containerID="cri-o://5773b9a4c0e3dea4a6c09a90c84f497e28b6b4a908181a25f61b8696df1aae30" gracePeriod=30 Oct 31 23:25:17.826708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.826672 7074 generic.go:356] "Generic (PLEG): container finished" podID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" containerID="5773b9a4c0e3dea4a6c09a90c84f497e28b6b4a908181a25f61b8696df1aae30" exitCode=0 Oct 31 23:25:17.826708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.826701 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" event={"ID":"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314","Type":"ContainerDied","Data":"5773b9a4c0e3dea4a6c09a90c84f497e28b6b4a908181a25f61b8696df1aae30"} Oct 31 23:25:17.947972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.947941 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:17.948175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.948010 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:17.976027 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:17.976003 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:25:18.088252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.087116 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\") pod \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " Oct 31 23:25:18.088252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.087197 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\") pod \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " Oct 31 23:25:18.088252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.087233 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\") pod \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " Oct 31 23:25:18.088252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.087325 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\" (UniqueName: \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\") pod \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\" (UID: \"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314\") " Oct 31 23:25:18.088605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.088367 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca" pod "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" (UID: "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:18.090346 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.090310 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv" pod "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" (UID: "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314"). InnerVolumeSpecName "kube-api-access-9pmlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:18.098091 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.090658 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs" pod "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" (UID: "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:18.100659 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.100628 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls" pod "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" (UID: "4183fdfd-22de-4bdb-9b13-f3bdc1bbe314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:18.149618 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.149579 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:25:18.150690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.150662 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c139df47-d3de-4818-805f-05810cdccb12" path="/var/lib/kubelet/pods/c139df47-d3de-4818-805f-05810cdccb12/volumes" Oct 31 23:25:18.151271 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.151251 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" path="/var/lib/kubelet/pods/e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9/volumes" Oct 31 23:25:18.190110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.190028 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.190110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.190063 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.190110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.190077 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.190110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.190089 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmlv\" (UniqueName: \"kubernetes.io/projected/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314-kube-api-access-9pmlv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.290708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290675 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.290887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290716 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.290887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290754 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.290887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290878 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290908 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.290962 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291045 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291079 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291111 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:18.291188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291143 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291216 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\") pod \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\" (UID: \"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656\") " Oct 31 23:25:18.291525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291319 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:18.291645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291574 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:18.291645 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291629 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:18.291958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291934 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.292149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291961 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.292149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.291978 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.292149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.292008 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.292149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.292023 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.293364 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.293339 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:18.293452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.293391 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:18.293452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.293409 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:18.293452 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.293420 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp" pod "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" (UID: "5a9a0210-fc3a-4c07-a5b0-1b15f2acc656"). InnerVolumeSpecName "kube-api-access-q8nvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:18.393318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.393284 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.393318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.393312 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.393318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.393322 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8nvp\" (UniqueName: \"kubernetes.io/projected/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-kube-api-access-q8nvp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.393791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.393333 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:18.625319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.625278 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:18.625319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.625311 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:18.625563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.625438 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:18.625563 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.625438 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:18.835327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.835290 7074 generic.go:356] "Generic (PLEG): container finished" podID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerID="9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253" exitCode=0 Oct 31 23:25:18.835512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.835343 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" Oct 31 23:25:18.835512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.835370 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" event={"ID":"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656","Type":"ContainerDied","Data":"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253"} Oct 31 23:25:18.835512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.835425 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6" event={"ID":"5a9a0210-fc3a-4c07-a5b0-1b15f2acc656","Type":"ContainerDied","Data":"7f667bca506938655fe0f7a35b8833ba69807704b5d53758922010ddefef4187"} Oct 31 23:25:18.835512 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.835454 7074 scope.go:122] "RemoveContainer" containerID="9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253" Oct 31 23:25:18.836967 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.836948 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" event={"ID":"4183fdfd-22de-4bdb-9b13-f3bdc1bbe314","Type":"ContainerDied","Data":"65bdcb3d10f34238b4ec085a2e6f7579038c55d8f5659c3090f83255eb59cbc1"} Oct 31 23:25:18.837064 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.836982 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j" Oct 31 23:25:18.847183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.847163 7074 scope.go:122] "RemoveContainer" containerID="9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253" Oct 31 23:25:18.847472 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:18.847453 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253\": container with ID starting with 9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253 not found: ID does not exist" containerID="9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253" Oct 31 23:25:18.847540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.847477 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253"} err="failed to get container status \"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253\": rpc error: code = NotFound desc = could not find container \"9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253\": container with ID starting with 9ef89c92c73333dd0a0c67cb0f753239b1792b7e4db6fe0a95a2fb3701c94253 not found: ID does not exist" Oct 31 23:25:18.847540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.847519 7074 scope.go:122] "RemoveContainer" containerID="5773b9a4c0e3dea4a6c09a90c84f497e28b6b4a908181a25f61b8696df1aae30" Oct 31 23:25:18.858628 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.858525 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:25:18.862278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.862254 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9e90c-54c676d85f-m467j"] Oct 31 23:25:18.877725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.877657 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:25:18.882564 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:18.882540 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-2kzk6"] Oct 31 23:25:19.335078 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.335042 7074 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:19.335225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.335110 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:19.587839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.573443 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:25:19.603190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.603161 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:25:19.708601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.708566 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\" (UniqueName: \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\") pod \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " Oct 31 23:25:19.708772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.708678 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\" (UniqueName: \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\") pod \"17e0bf96-8ee5-4334-a767-f69061825885\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " Oct 31 23:25:19.708772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.708745 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\") pod \"17e0bf96-8ee5-4334-a767-f69061825885\" (UID: \"17e0bf96-8ee5-4334-a767-f69061825885\") " Oct 31 23:25:19.708849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.708773 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\") pod \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\" (UID: \"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04\") " Oct 31 23:25:19.709191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.709160 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume" pod "17e0bf96-8ee5-4334-a767-f69061825885" (UID: "17e0bf96-8ee5-4334-a767-f69061825885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:19.709296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.709187 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume" pod "f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" (UID: "f00fd71e-9c7a-4b34-8bd2-cad937a6ab04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:19.709377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.709358 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e0bf96-8ee5-4334-a767-f69061825885-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:19.709425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.709383 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:19.710852 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.710830 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb" pod "f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" (UID: "f00fd71e-9c7a-4b34-8bd2-cad937a6ab04"). InnerVolumeSpecName "kube-api-access-sbncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:19.710938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.710856 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp" pod "17e0bf96-8ee5-4334-a767-f69061825885" (UID: "17e0bf96-8ee5-4334-a767-f69061825885"). InnerVolumeSpecName "kube-api-access-m55cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:19.810505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.810468 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbncb\" (UniqueName: \"kubernetes.io/projected/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04-kube-api-access-sbncb\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:19.810505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.810501 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55cp\" (UniqueName: \"kubernetes.io/projected/17e0bf96-8ee5-4334-a767-f69061825885-kube-api-access-m55cp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:19.846744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.846639 7074 generic.go:356] "Generic (PLEG): container finished" podID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerID="58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b" exitCode=0 Oct 31 23:25:19.846744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.846696 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" Oct 31 23:25:19.846744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.846694 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" event={"ID":"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04","Type":"ContainerDied","Data":"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b"} Oct 31 23:25:19.847048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.846757 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz" event={"ID":"f00fd71e-9c7a-4b34-8bd2-cad937a6ab04","Type":"ContainerDied","Data":"f8cfcc65e6942dd55a44a626c40e531c7b519eb6f6a02965c6bf29bd7c582d6a"} Oct 31 23:25:19.847048 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.846789 7074 scope.go:122] "RemoveContainer" containerID="58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b" Oct 31 23:25:19.848923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.848896 7074 generic.go:356] "Generic (PLEG): container finished" podID="17e0bf96-8ee5-4334-a767-f69061825885" containerID="2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc" exitCode=0 Oct 31 23:25:19.849100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.848961 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" Oct 31 23:25:19.849100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.848982 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" event={"ID":"17e0bf96-8ee5-4334-a767-f69061825885","Type":"ContainerDied","Data":"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc"} Oct 31 23:25:19.849100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.849081 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg" event={"ID":"17e0bf96-8ee5-4334-a767-f69061825885","Type":"ContainerDied","Data":"4edc3ab7f1c1a708f8dd507ba5ce089b307ca0d30f9424f73495fb95cb80caf8"} Oct 31 23:25:19.859564 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.859546 7074 scope.go:122] "RemoveContainer" containerID="58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b" Oct 31 23:25:19.859856 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:19.859833 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b\": container with ID starting with 58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b not found: ID does not exist" containerID="58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b" Oct 31 23:25:19.859893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.859865 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b"} err="failed to get container status \"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b\": rpc error: code = NotFound desc = could not find container \"58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b\": container with ID starting with 58b64b823fed93de901a4752e39c960ab265c65ec75be350919f6cce2f5d977b not found: ID does not exist" Oct 31 23:25:19.859893 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.859882 7074 scope.go:122] "RemoveContainer" containerID="2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc" Oct 31 23:25:19.870688 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.870665 7074 scope.go:122] "RemoveContainer" containerID="2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc" Oct 31 23:25:19.871081 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:19.871045 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc\": container with ID starting with 2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc not found: ID does not exist" containerID="2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc" Oct 31 23:25:19.871171 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.871083 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc"} err="failed to get container status \"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc\": rpc error: code = NotFound desc = could not find container \"2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc\": container with ID starting with 2291b7086699fc3e36a96923021809c644d162589a703f4d986ff0d970a7c9fc not found: ID does not exist" Oct 31 23:25:19.880897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.880867 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:25:19.884659 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.884630 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-67fce9-7b88689c5-g9nkz"] Oct 31 23:25:19.896653 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.896623 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:25:19.901134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:19.901109 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-67fce9-7cfffb9fb4-t9cfg"] Oct 31 23:25:20.143226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.143144 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e0bf96-8ee5-4334-a767-f69061825885" path="/var/lib/kubelet/pods/17e0bf96-8ee5-4334-a767-f69061825885/volumes" Oct 31 23:25:20.143758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.143735 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" path="/var/lib/kubelet/pods/4183fdfd-22de-4bdb-9b13-f3bdc1bbe314/volumes" Oct 31 23:25:20.144260 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.144240 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" path="/var/lib/kubelet/pods/5a9a0210-fc3a-4c07-a5b0-1b15f2acc656/volumes" Oct 31 23:25:20.145029 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.144982 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" path="/var/lib/kubelet/pods/f00fd71e-9c7a-4b34-8bd2-cad937a6ab04/volumes" Oct 31 23:25:20.235240 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.235202 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:25:20.235530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.235442 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" containerID="cri-o://25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5" gracePeriod=30 Oct 31 23:25:20.243144 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.240797 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:25:20.243318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.243198 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:25:20.243416 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.243390 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" containerID="cri-o://053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c" gracePeriod=30 Oct 31 23:25:20.244572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:20.243750 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" containerID="cri-o://f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359" gracePeriod=30 Oct 31 23:25:22.547970 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.547936 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:25:22.553712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.553684 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:25:22.599055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.598231 7074 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:22.599055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.598305 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:22.686830 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.686801 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:25:22.725476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.725451 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:25:22.754682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754643 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754688 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754769 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754795 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754831 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754872 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.754949 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.754928 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.755265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755056 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.755265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755072 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.755384 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755299 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.755491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755450 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:22.755747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755643 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.755836 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.755790 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\") pod \"ccfea965-40cb-411b-9aa4-4ca947983ed2\" (UID: \"ccfea965-40cb-411b-9aa4-4ca947983ed2\") " Oct 31 23:25:22.756053 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756002 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.756467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756442 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.756579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756472 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.756579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756488 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.756579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756503 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfea965-40cb-411b-9aa4-4ca947983ed2-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.756579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.756517 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.758548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.758147 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.758548 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.758500 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "kube-api-access-n7jbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.758955 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.758931 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:22.759172 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.759134 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy" pod "ccfea965-40cb-411b-9aa4-4ca947983ed2" (UID: "ccfea965-40cb-411b-9aa4-4ca947983ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.857101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857065 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857123 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857209 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857254 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857314 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857350 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857390 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857402 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857432 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857489 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857478 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\") pod \"97a5197a-1b71-42ea-ae3d-705d0396704a\" (UID: \"97a5197a-1b71-42ea-ae3d-705d0396704a\") " Oct 31 23:25:22.857804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857502 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.857804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857602 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.857804 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.857726 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:22.858189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858157 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858171 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858217 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858234 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858251 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858266 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a5197a-1b71-42ea-ae3d-705d0396704a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858283 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858308 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858299 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfea965-40cb-411b-9aa4-4ca947983ed2-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.858680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.858313 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jbs\" (UniqueName: \"kubernetes.io/projected/ccfea965-40cb-411b-9aa4-4ca947983ed2-kube-api-access-n7jbs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.859946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.859905 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.860092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.860066 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "kube-api-access-t8kfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.860281 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.860260 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:22.860355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.860270 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy" pod "97a5197a-1b71-42ea-ae3d-705d0396704a" (UID: "97a5197a-1b71-42ea-ae3d-705d0396704a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.877882 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.877857 7074 generic.go:356] "Generic (PLEG): container finished" podID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" exitCode=0 Oct 31 23:25:22.878059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.877910 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" Oct 31 23:25:22.878059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.877950 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" event={"ID":"ccfea965-40cb-411b-9aa4-4ca947983ed2","Type":"ContainerDied","Data":"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2"} Oct 31 23:25:22.878059 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.878050 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj" event={"ID":"ccfea965-40cb-411b-9aa4-4ca947983ed2","Type":"ContainerDied","Data":"964976912a993a08c3afa6cb6fe1571c17de41400a5ff90e4b3fe0ff27a4ad56"} Oct 31 23:25:22.878250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.878086 7074 scope.go:122] "RemoveContainer" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" Oct 31 23:25:22.879782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.879728 7074 generic.go:356] "Generic (PLEG): container finished" podID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" exitCode=0 Oct 31 23:25:22.879782 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.879775 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" Oct 31 23:25:22.879937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.879916 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" event={"ID":"97a5197a-1b71-42ea-ae3d-705d0396704a","Type":"ContainerDied","Data":"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1"} Oct 31 23:25:22.880004 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.879954 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2" event={"ID":"97a5197a-1b71-42ea-ae3d-705d0396704a","Type":"ContainerDied","Data":"46e1c3aaacd4ecc4772172c72d5c0fa7aea6823af425307e203bc97619d42785"} Oct 31 23:25:22.886296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.886275 7074 scope.go:122] "RemoveContainer" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.892952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.892928 7074 scope.go:122] "RemoveContainer" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" Oct 31 23:25:22.893376 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:22.893355 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2\": container with ID starting with e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2 not found: ID does not exist" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" Oct 31 23:25:22.893475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.893381 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2"} err="failed to get container status \"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2\": rpc error: code = NotFound desc = could not find container \"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2\": container with ID starting with e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2 not found: ID does not exist" Oct 31 23:25:22.893475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.893398 7074 scope.go:122] "RemoveContainer" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.898501 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.898479 7074 scope.go:122] "RemoveContainer" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.898794 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:22.898764 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1\": container with ID starting with 956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1 not found: ID does not exist" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.898883 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.898788 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1"} err="failed to get container status \"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1\": rpc error: code = NotFound desc = could not find container \"956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1\": container with ID starting with 956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1 not found: ID does not exist" Oct 31 23:25:22.898957 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:22.898882 7074 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2_openshift-ingress_97a5197a-1b71-42ea-ae3d-705d0396704a_0 in pod sandbox 46e1c3aaacd4ecc4772172c72d5c0fa7aea6823af425307e203bc97619d42785 from index: no such id: '956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1'" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.898957 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:22.898903 7074 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2_openshift-ingress_97a5197a-1b71-42ea-ae3d-705d0396704a_0 in pod sandbox 46e1c3aaacd4ecc4772172c72d5c0fa7aea6823af425307e203bc97619d42785 from index: no such id: '956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1'" containerID="956b5e50a2a88bcc54c8549f84c38086263d2a0f18d9ff839883243d0ee1cdf1" Oct 31 23:25:22.898957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.898926 7074 scope.go:122] "RemoveContainer" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" Oct 31 23:25:22.899216 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:22.899193 7074 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2\": rpc error: code = NotFound desc = could not find container \"e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2\": container with ID starting with e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2 not found: ID does not exist" containerID="e39906775f4409380b178c84260a1bd4e3455ec4fcbca5364d1890b09816e6b2" Oct 31 23:25:22.927758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.927730 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:25:22.941984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.941954 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-gmdmj"] Oct 31 23:25:22.959129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959105 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.959274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959136 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.959274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959153 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.959274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959168 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kfn\" (UniqueName: \"kubernetes.io/projected/97a5197a-1b71-42ea-ae3d-705d0396704a-kube-api-access-t8kfn\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.959274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959182 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a5197a-1b71-42ea-ae3d-705d0396704a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.959457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.959367 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:25:22.966154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:22.966132 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-l5vd2"] Oct 31 23:25:24.142593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:24.142554 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" path="/var/lib/kubelet/pods/97a5197a-1b71-42ea-ae3d-705d0396704a/volumes" Oct 31 23:25:24.143079 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:24.143064 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" path="/var/lib/kubelet/pods/ccfea965-40cb-411b-9aa4-4ca947983ed2/volumes" Oct 31 23:25:25.538643 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.538606 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:25:25.633129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.633098 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:25:25.638376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.638351 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:25:25.698439 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698400 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.698636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698470 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.698636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698504 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.698636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698571 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.698636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698613 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.698876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698690 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.698876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698712 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.699003 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.698942 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.699088 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699060 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:25.699172 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699108 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.699172 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699148 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.699274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699196 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\") pod \"4421fe12-f738-4e07-9463-a15933118db0\" (UID: \"4421fe12-f738-4e07-9463-a15933118db0\") " Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699799 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699908 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699929 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4421fe12-f738-4e07-9463-a15933118db0-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699945 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.699958 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.700041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.700020 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.703405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.703377 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.703531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.703406 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "kube-api-access-tpvrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.704690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.704664 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:25.705118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.705095 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token" pod "4421fe12-f738-4e07-9463-a15933118db0" (UID: "4421fe12-f738-4e07-9463-a15933118db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.800980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.800893 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.800980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.800951 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.800980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.800971 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801040 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801063 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801095 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801134 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801203 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801233 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801240 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801255 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.801315 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801287 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801373 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801518 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801526 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801536 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801615 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801630 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801689 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801717 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801747 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801789 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801828 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\") pod \"2ac96af3-4543-4010-a8c3-616ac77264da\" (UID: \"2ac96af3-4543-4010-a8c3-616ac77264da\") " Oct 31 23:25:25.802092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.801882 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\") pod \"c6662d9f-823a-4d16-bf62-86dc28511ef7\" (UID: \"c6662d9f-823a-4d16-bf62-86dc28511ef7\") " Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802383 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4421fe12-f738-4e07-9463-a15933118db0-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802392 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802399 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802407 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvrh\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-kube-api-access-tpvrh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802445 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802461 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802475 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4421fe12-f738-4e07-9463-a15933118db0-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802490 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802504 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4421fe12-f738-4e07-9463-a15933118db0-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802519 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802533 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802646 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac96af3-4543-4010-a8c3-616ac77264da-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802662 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802588 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.802750 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.802619 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.803801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.803770 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:25.804067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.804048 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.804278 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.804225 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "kube-api-access-cvffh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.804366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.804331 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.804366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.804336 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.804595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.804578 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc" pod "c6662d9f-823a-4d16-bf62-86dc28511ef7" (UID: "c6662d9f-823a-4d16-bf62-86dc28511ef7"). InnerVolumeSpecName "kube-api-access-76cgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.805256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.805238 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.805540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.805526 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo" pod "2ac96af3-4543-4010-a8c3-616ac77264da" (UID: "2ac96af3-4543-4010-a8c3-616ac77264da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:25.903540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903501 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac96af3-4543-4010-a8c3-616ac77264da-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903538 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903554 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6662d9f-823a-4d16-bf62-86dc28511ef7-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903569 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903582 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac96af3-4543-4010-a8c3-616ac77264da-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903608 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903622 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903636 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903649 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903664 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvffh\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-kube-api-access-cvffh\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903680 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac96af3-4543-4010-a8c3-616ac77264da-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903677 7074 generic.go:356] "Generic (PLEG): container finished" podID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerID="25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5" exitCode=0 Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903694 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cgc\" (UniqueName: \"kubernetes.io/projected/c6662d9f-823a-4d16-bf62-86dc28511ef7-kube-api-access-76cgc\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903729 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" Oct 31 23:25:25.903812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903756 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" event={"ID":"c6662d9f-823a-4d16-bf62-86dc28511ef7","Type":"ContainerDied","Data":"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5"} Oct 31 23:25:25.904403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903842 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg" event={"ID":"c6662d9f-823a-4d16-bf62-86dc28511ef7","Type":"ContainerDied","Data":"c557e18d48bf0dcb0e791f0a9729dd313d809dcc88b05c38fafc135209eb5495"} Oct 31 23:25:25.904403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.903860 7074 scope.go:122] "RemoveContainer" containerID="25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5" Oct 31 23:25:25.905395 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.905370 7074 generic.go:356] "Generic (PLEG): container finished" podID="4421fe12-f738-4e07-9463-a15933118db0" containerID="053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c" exitCode=0 Oct 31 23:25:25.905505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.905419 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" Oct 31 23:25:25.905505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.905492 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" event={"ID":"4421fe12-f738-4e07-9463-a15933118db0","Type":"ContainerDied","Data":"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c"} Oct 31 23:25:25.905621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.905532 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr" event={"ID":"4421fe12-f738-4e07-9463-a15933118db0","Type":"ContainerDied","Data":"af00602279314a65473773985eea8b6e8d31ba2ea40865f4b7ab466fbd87e38a"} Oct 31 23:25:25.908665 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.908639 7074 generic.go:356] "Generic (PLEG): container finished" podID="2ac96af3-4543-4010-a8c3-616ac77264da" containerID="f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359" exitCode=0 Oct 31 23:25:25.908762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.908673 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" event={"ID":"2ac96af3-4543-4010-a8c3-616ac77264da","Type":"ContainerDied","Data":"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359"} Oct 31 23:25:25.908762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.908687 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" Oct 31 23:25:25.908762 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.908723 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt" event={"ID":"2ac96af3-4543-4010-a8c3-616ac77264da","Type":"ContainerDied","Data":"e84f0044341866064dea4ebb1caecf7071c6c4a57f4ca2ca356051a14d8cff6d"} Oct 31 23:25:25.916147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.916124 7074 scope.go:122] "RemoveContainer" containerID="25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5" Oct 31 23:25:25.916530 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:25.916503 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5\": container with ID starting with 25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5 not found: ID does not exist" containerID="25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5" Oct 31 23:25:25.916632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.916540 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5"} err="failed to get container status \"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5\": rpc error: code = NotFound desc = could not find container \"25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5\": container with ID starting with 25dad48d11d0461a355963b40119a9cc01614726c2653563f30253b1fb1af5a5 not found: ID does not exist" Oct 31 23:25:25.916632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.916565 7074 scope.go:122] "RemoveContainer" containerID="053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c" Oct 31 23:25:25.928377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.928354 7074 scope.go:122] "RemoveContainer" containerID="053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c" Oct 31 23:25:25.928707 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:25.928690 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c\": container with ID starting with 053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c not found: ID does not exist" containerID="053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c" Oct 31 23:25:25.928763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.928714 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c"} err="failed to get container status \"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c\": rpc error: code = NotFound desc = could not find container \"053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c\": container with ID starting with 053166a8715312c9fc7e57e68508959c5cc6ab84715b060cbcf50feac630941c not found: ID does not exist" Oct 31 23:25:25.928763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.928732 7074 scope.go:122] "RemoveContainer" containerID="f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359" Oct 31 23:25:25.936753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.936725 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:25:25.939462 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.939442 7074 scope.go:122] "RemoveContainer" containerID="f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359" Oct 31 23:25:25.939773 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:25.939755 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359\": container with ID starting with f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359 not found: ID does not exist" containerID="f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359" Oct 31 23:25:25.939833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.939782 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359"} err="failed to get container status \"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359\": rpc error: code = NotFound desc = could not find container \"f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359\": container with ID starting with f0308dcd2f9ed2260f72953a53fd7a10cec6fc939615cc828b0052a102ff0359 not found: ID does not exist" Oct 31 23:25:25.940484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.940462 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dlscg"] Oct 31 23:25:25.955184 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.955158 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:25:25.960980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.960954 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-tsrkr"] Oct 31 23:25:25.973390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.973361 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:25:25.979255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:25.979230 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-p4cjt"] Oct 31 23:25:26.146966 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.146889 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" path="/var/lib/kubelet/pods/2ac96af3-4543-4010-a8c3-616ac77264da/volumes" Oct 31 23:25:26.147626 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.147606 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4421fe12-f738-4e07-9463-a15933118db0" path="/var/lib/kubelet/pods/4421fe12-f738-4e07-9463-a15933118db0/volumes" Oct 31 23:25:26.148275 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.148255 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" path="/var/lib/kubelet/pods/c6662d9f-823a-4d16-bf62-86dc28511ef7/volumes" Oct 31 23:25:26.717871 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.717839 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:25:26.721684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.721657 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:25:26.814965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.814929 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\" (UniqueName: \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\") pod \"a779def2-64c5-4553-b53a-0ff1564605ab\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " Oct 31 23:25:26.815170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815040 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\") pod \"a779def2-64c5-4553-b53a-0ff1564605ab\" (UID: \"a779def2-64c5-4553-b53a-0ff1564605ab\") " Oct 31 23:25:26.815170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815086 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\") pod \"80dc3108-a881-484a-bda6-ccc8f9374bbe\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " Oct 31 23:25:26.815265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815208 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\" (UniqueName: \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\") pod \"80dc3108-a881-484a-bda6-ccc8f9374bbe\" (UID: \"80dc3108-a881-484a-bda6-ccc8f9374bbe\") " Oct 31 23:25:26.815514 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815476 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume" pod "a779def2-64c5-4553-b53a-0ff1564605ab" (UID: "a779def2-64c5-4553-b53a-0ff1564605ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:26.815514 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815487 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume" pod "80dc3108-a881-484a-bda6-ccc8f9374bbe" (UID: "80dc3108-a881-484a-bda6-ccc8f9374bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:26.815662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815642 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779def2-64c5-4553-b53a-0ff1564605ab-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:26.815716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.815663 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dc3108-a881-484a-bda6-ccc8f9374bbe-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:26.817155 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.817135 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs" pod "a779def2-64c5-4553-b53a-0ff1564605ab" (UID: "a779def2-64c5-4553-b53a-0ff1564605ab"). InnerVolumeSpecName "kube-api-access-qb6cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:26.817210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.817166 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw" pod "80dc3108-a881-484a-bda6-ccc8f9374bbe" (UID: "80dc3108-a881-484a-bda6-ccc8f9374bbe"). InnerVolumeSpecName "kube-api-access-ncwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:26.915583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.915548 7074 generic.go:356] "Generic (PLEG): container finished" podID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerID="e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1" exitCode=0 Oct 31 23:25:26.915759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.915610 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" Oct 31 23:25:26.915759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.915638 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" event={"ID":"80dc3108-a881-484a-bda6-ccc8f9374bbe","Type":"ContainerDied","Data":"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1"} Oct 31 23:25:26.915759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.915692 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs" event={"ID":"80dc3108-a881-484a-bda6-ccc8f9374bbe","Type":"ContainerDied","Data":"54d77c652439ceb5095476b89c5d8ce4646dcb3edd9de8f4fe5202108962ba42"} Oct 31 23:25:26.915759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.915724 7074 scope.go:122] "RemoveContainer" containerID="e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1" Oct 31 23:25:26.916523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.916499 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwgw\" (UniqueName: \"kubernetes.io/projected/80dc3108-a881-484a-bda6-ccc8f9374bbe-kube-api-access-ncwgw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:26.916523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.916525 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6cs\" (UniqueName: \"kubernetes.io/projected/a779def2-64c5-4553-b53a-0ff1564605ab-kube-api-access-qb6cs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:26.920460 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.920439 7074 generic.go:356] "Generic (PLEG): container finished" podID="a779def2-64c5-4553-b53a-0ff1564605ab" containerID="d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416" exitCode=0 Oct 31 23:25:26.920565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.920474 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" event={"ID":"a779def2-64c5-4553-b53a-0ff1564605ab","Type":"ContainerDied","Data":"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416"} Oct 31 23:25:26.920565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.920483 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" Oct 31 23:25:26.920565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.920509 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n" event={"ID":"a779def2-64c5-4553-b53a-0ff1564605ab","Type":"ContainerDied","Data":"045848c6e028ff8ead085bf12e6ea797508c1b02d557dc1ffe8dab4a55a8f12a"} Oct 31 23:25:26.926124 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.926109 7074 scope.go:122] "RemoveContainer" containerID="e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1" Oct 31 23:25:26.926443 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:26.926426 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1\": container with ID starting with e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1 not found: ID does not exist" containerID="e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1" Oct 31 23:25:26.926491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.926451 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1"} err="failed to get container status \"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1\": rpc error: code = NotFound desc = could not find container \"e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1\": container with ID starting with e941116d66017ed935b800917b31789bf96c49e07c928f2a3caaf75d8d0aebc1 not found: ID does not exist" Oct 31 23:25:26.926491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.926467 7074 scope.go:122] "RemoveContainer" containerID="d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416" Oct 31 23:25:26.935377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.935359 7074 scope.go:122] "RemoveContainer" containerID="d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416" Oct 31 23:25:26.935652 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:26.935637 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416\": container with ID starting with d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416 not found: ID does not exist" containerID="d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416" Oct 31 23:25:26.935713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.935657 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416"} err="failed to get container status \"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416\": rpc error: code = NotFound desc = could not find container \"d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416\": container with ID starting with d8465511dcf93797ce17c4faa16cbc9e84dd2373232bb5c1fd8820d7aff7a416 not found: ID does not exist" Oct 31 23:25:26.939840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.939819 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:25:26.945332 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.945312 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04c5c2-846bcb4fc8-kclcs"] Oct 31 23:25:26.957205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.957184 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:25:26.961139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:26.961119 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04c5c2-656c7b899f-q225n"] Oct 31 23:25:28.141608 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:28.141577 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" path="/var/lib/kubelet/pods/80dc3108-a881-484a-bda6-ccc8f9374bbe/volumes" Oct 31 23:25:28.142016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:28.141977 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" path="/var/lib/kubelet/pods/a779def2-64c5-4553-b53a-0ff1564605ab/volumes" Oct 31 23:25:30.972044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:30.972010 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:25:30.977057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:30.977031 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:25:30.985544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:30.985515 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:25:36.261580 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:36.261543 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:25:36.279396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:36.279359 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:25:41.262657 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:41.262594 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" containerID="cri-o://b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485" gracePeriod=25 Oct 31 23:25:41.280067 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:41.280034 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" containerID="cri-o://3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb" gracePeriod=25 Oct 31 23:25:44.888354 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:44.888304 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:25:45.995984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:45.995941 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" containerID="cri-o://4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed" gracePeriod=45 Oct 31 23:25:45.998635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:45.998606 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" containerID="cri-o://81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479" gracePeriod=45 Oct 31 23:25:46.006262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.006225 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" containerID="cri-o://d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a" gracePeriod=45 Oct 31 23:25:46.281744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.281717 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:25:46.304321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.304262 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:25:46.311575 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.311536 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:25:46.442557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442513 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.442557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442559 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.442840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442601 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\" (UniqueName: \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.442840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442633 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.442840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442667 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\" (UniqueName: \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.442840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442753 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.442840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442838 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442866 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442895 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442894 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.442927 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443050 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443103 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.443162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443133 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443178 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443194 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443223 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443262 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\") pod \"e12de15d-7208-4703-a91d-ef282c36e0fb\" (UID: \"e12de15d-7208-4703-a91d-ef282c36e0fb\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443294 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443348 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\" (UniqueName: \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\") pod \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\" (UID: \"3d48f7b7-120b-4452-9c04-121ffc7ef7ef\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443388 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\") pod \"d45f0f22-b29b-470e-8e69-57e1298605e6\" (UID: \"d45f0f22-b29b-470e-8e69-57e1298605e6\") " Oct 31 23:25:46.443555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443406 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.443954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443581 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.443954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443901 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.443954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443922 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.443954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443940 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.444192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443955 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.444192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.443969 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.444309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.444216 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.444411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.444391 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.444615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.444574 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.444867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.444846 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.445565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.445534 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "kube-api-access-2hx2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:46.445670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.445630 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.445737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.445667 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "kube-api-access-x6qcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:46.445933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.445914 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:46.446026 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.445925 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:46.446330 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.446309 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.446381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.446323 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm" pod "d45f0f22-b29b-470e-8e69-57e1298605e6" (UID: "d45f0f22-b29b-470e-8e69-57e1298605e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:46.447045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.447027 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs" pod "e12de15d-7208-4703-a91d-ef282c36e0fb" (UID: "e12de15d-7208-4703-a91d-ef282c36e0fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:46.447543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.447527 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg" pod "3d48f7b7-120b-4452-9c04-121ffc7ef7ef" (UID: "3d48f7b7-120b-4452-9c04-121ffc7ef7ef"). InnerVolumeSpecName "kube-api-access-8xdcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545415 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545445 7074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-model-cache\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545455 7074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-home\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545464 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12de15d-7208-4703-a91d-ef282c36e0fb-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545472 7074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-tmp-dir\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545482 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xdcg\" (UniqueName: \"kubernetes.io/projected/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-kube-api-access-8xdcg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545491 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45f0f22-b29b-470e-8e69-57e1298605e6-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545499 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hx2c\" (UniqueName: \"kubernetes.io/projected/d45f0f22-b29b-470e-8e69-57e1298605e6-kube-api-access-2hx2c\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545510 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12de15d-7208-4703-a91d-ef282c36e0fb-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545518 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6qcw\" (UniqueName: \"kubernetes.io/projected/e12de15d-7208-4703-a91d-ef282c36e0fb-kube-api-access-x6qcw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545526 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545535 7074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d48f7b7-120b-4452-9c04-121ffc7ef7ef-tls-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:46.545867 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:46.545542 7074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45f0f22-b29b-470e-8e69-57e1298605e6-dshm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:47.055634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.055597 7074 generic.go:356] "Generic (PLEG): container finished" podID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerID="4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed" exitCode=0 Oct 31 23:25:47.056115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.055628 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" event={"ID":"d45f0f22-b29b-470e-8e69-57e1298605e6","Type":"ContainerDied","Data":"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed"} Oct 31 23:25:47.056115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.055650 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" Oct 31 23:25:47.056115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.055682 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj" event={"ID":"d45f0f22-b29b-470e-8e69-57e1298605e6","Type":"ContainerDied","Data":"d060ad9dc8e41d6fafac54c1998f10fc88d02cbb92811e541951d506ec8fa384"} Oct 31 23:25:47.056115 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.055708 7074 scope.go:122] "RemoveContainer" containerID="4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed" Oct 31 23:25:47.057165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.057148 7074 generic.go:356] "Generic (PLEG): container finished" podID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerID="d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a" exitCode=0 Oct 31 23:25:47.057237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.057182 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" Oct 31 23:25:47.057237 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.057226 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" event={"ID":"3d48f7b7-120b-4452-9c04-121ffc7ef7ef","Type":"ContainerDied","Data":"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a"} Oct 31 23:25:47.057358 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.057260 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz" event={"ID":"3d48f7b7-120b-4452-9c04-121ffc7ef7ef","Type":"ContainerDied","Data":"d42d75f280ef3f09210613ab8939cb62b5a8aa4d8955709301606c66fccfc821"} Oct 31 23:25:47.058845 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.058817 7074 generic.go:356] "Generic (PLEG): container finished" podID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerID="81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479" exitCode=0 Oct 31 23:25:47.058938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.058841 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" event={"ID":"e12de15d-7208-4703-a91d-ef282c36e0fb","Type":"ContainerDied","Data":"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479"} Oct 31 23:25:47.058938 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.058873 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" Oct 31 23:25:47.059046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.058879 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk" event={"ID":"e12de15d-7208-4703-a91d-ef282c36e0fb","Type":"ContainerDied","Data":"1b397cc806b4f26ddfd6a01cf2a6c49061f394a2dc8c2f49da93d3311727ee47"} Oct 31 23:25:47.070633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.070609 7074 scope.go:122] "RemoveContainer" containerID="4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed" Oct 31 23:25:47.070958 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:47.070941 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed\": container with ID starting with 4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed not found: ID does not exist" containerID="4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed" Oct 31 23:25:47.071050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.070968 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed"} err="failed to get container status \"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed\": rpc error: code = NotFound desc = could not find container \"4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed\": container with ID starting with 4e5f426c36670946f36a09cb7b8133588d02b3b62a88daaa47b12a92592301ed not found: ID does not exist" Oct 31 23:25:47.071050 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.070984 7074 scope.go:122] "RemoveContainer" containerID="d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a" Oct 31 23:25:47.080518 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.080499 7074 scope.go:122] "RemoveContainer" containerID="d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a" Oct 31 23:25:47.080820 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:47.080795 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a\": container with ID starting with d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a not found: ID does not exist" containerID="d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a" Oct 31 23:25:47.080874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.080828 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a"} err="failed to get container status \"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a\": rpc error: code = NotFound desc = could not find container \"d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a\": container with ID starting with d6dfe36aabc361d486809d221efb92bab54adc0829ae9f641d2219a86ac6cb7a not found: ID does not exist" Oct 31 23:25:47.080874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.080845 7074 scope.go:122] "RemoveContainer" containerID="81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479" Oct 31 23:25:47.089173 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.089148 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:25:47.090117 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.090100 7074 scope.go:122] "RemoveContainer" containerID="81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479" Oct 31 23:25:47.090437 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:47.090417 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479\": container with ID starting with 81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479 not found: ID does not exist" containerID="81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479" Oct 31 23:25:47.090483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.090444 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479"} err="failed to get container status \"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479\": rpc error: code = NotFound desc = could not find container \"81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479\": container with ID starting with 81b332efda6f4a8fe6c4309d49f7adeab467b39fde6832e03d5a7e247d56d479 not found: ID does not exist" Oct 31 23:25:47.099002 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.098962 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-255748/premium-w2-255748-kserve-549db6f8f5-5lndz"] Oct 31 23:25:47.111157 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.111129 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:25:47.116800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.116773 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-2-w2-255748-kserve-7c9dddd5d9-fw7cj"] Oct 31 23:25:47.128430 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.128393 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:25:47.132615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:47.132590 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-255748/distinct-w2-255748-kserve-c7449b854-ntbjk"] Oct 31 23:25:48.142035 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:48.141973 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" path="/var/lib/kubelet/pods/3d48f7b7-120b-4452-9c04-121ffc7ef7ef/volumes" Oct 31 23:25:48.142463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:48.142448 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" path="/var/lib/kubelet/pods/d45f0f22-b29b-470e-8e69-57e1298605e6/volumes" Oct 31 23:25:48.142868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:48.142855 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" path="/var/lib/kubelet/pods/e12de15d-7208-4703-a91d-ef282c36e0fb/volumes" Oct 31 23:25:48.623687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:48.623651 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:48.623687 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:48.623684 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:52.182473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.182436 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:25:52.182984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.182702 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" podUID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" containerName="maas-api" containerID="cri-o://33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378" gracePeriod=30 Oct 31 23:25:52.483607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.483564 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:25:52.614458 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.614414 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\" (UniqueName: \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\") pod \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " Oct 31 23:25:52.614639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.614606 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\") pod \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " Oct 31 23:25:52.614720 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.614664 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\") pod \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " Oct 31 23:25:52.614853 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.614732 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\") pod \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\" (UID: \"9352f587-17c5-4cb8-a7b3-0e5a517e882f\") " Oct 31 23:25:52.615415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.615376 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca" pod "9352f587-17c5-4cb8-a7b3-0e5a517e882f" (UID: "9352f587-17c5-4cb8-a7b3-0e5a517e882f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:52.617279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.617250 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls" pod "9352f587-17c5-4cb8-a7b3-0e5a517e882f" (UID: "9352f587-17c5-4cb8-a7b3-0e5a517e882f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:52.617581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.617539 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs" pod "9352f587-17c5-4cb8-a7b3-0e5a517e882f" (UID: "9352f587-17c5-4cb8-a7b3-0e5a517e882f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:52.619160 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.619132 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92" pod "9352f587-17c5-4cb8-a7b3-0e5a517e882f" (UID: "9352f587-17c5-4cb8-a7b3-0e5a517e882f"). InnerVolumeSpecName "kube-api-access-9jg92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:52.716759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.716675 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-maas-api-tls\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:52.716759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.716704 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9352f587-17c5-4cb8-a7b3-0e5a517e882f-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:52.716759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.716715 7074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9352f587-17c5-4cb8-a7b3-0e5a517e882f-openshift-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:52.716759 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:52.716724 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jg92\" (UniqueName: \"kubernetes.io/projected/9352f587-17c5-4cb8-a7b3-0e5a517e882f-kube-api-access-9jg92\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:53.098204 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.098159 7074 generic.go:356] "Generic (PLEG): container finished" podID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" containerID="33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378" exitCode=0 Oct 31 23:25:53.098427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.098220 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" Oct 31 23:25:53.098427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.098246 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" event={"ID":"9352f587-17c5-4cb8-a7b3-0e5a517e882f","Type":"ContainerDied","Data":"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378"} Oct 31 23:25:53.098427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.098305 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk" event={"ID":"9352f587-17c5-4cb8-a7b3-0e5a517e882f","Type":"ContainerDied","Data":"fac635448d01d0d2abeb6118974b2bec385038ce5529982fe3036741d5505754"} Oct 31 23:25:53.098427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.098327 7074 scope.go:122] "RemoveContainer" containerID="33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378" Oct 31 23:25:53.111891 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.109665 7074 scope.go:122] "RemoveContainer" containerID="33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378" Oct 31 23:25:53.112450 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:25:53.112426 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378\": container with ID starting with 33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378 not found: ID does not exist" containerID="33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378" Oct 31 23:25:53.112559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.112455 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378"} err="failed to get container status \"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378\": rpc error: code = NotFound desc = could not find container \"33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378\": container with ID starting with 33d0f3fb6d05a6a133d4f990a15bafda18c1533be2073fe0f116103342c8b378 not found: ID does not exist" Oct 31 23:25:53.123210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.123187 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:25:53.127385 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:53.127360 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-255748-85bb494466-sv4dk"] Oct 31 23:25:54.140623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:54.140589 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" path="/var/lib/kubelet/pods/9352f587-17c5-4cb8-a7b3-0e5a517e882f/volumes" Oct 31 23:25:58.623599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:58.623551 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:58.623599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:58.623572 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:25:59.471736 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:59.471697 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:25:59.472030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:25:59.471965 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" containerID="cri-o://10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c" gracePeriod=30 Oct 31 23:26:01.534055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.533972 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:26:01.561559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.561530 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:26:01.608820 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.608787 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\" (UniqueName: \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\") pod \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " Oct 31 23:26:01.609023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.608899 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\") pod \"dd605101-22ca-4801-bc41-2f734374bc8f\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " Oct 31 23:26:01.609023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.608928 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\" (UniqueName: \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\") pod \"dd605101-22ca-4801-bc41-2f734374bc8f\" (UID: \"dd605101-22ca-4801-bc41-2f734374bc8f\") " Oct 31 23:26:01.609023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.609019 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\") pod \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\" (UID: \"b787ffbb-5157-475b-8eb2-b7c6f5e8671d\") " Oct 31 23:26:01.609366 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.609341 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume" pod "dd605101-22ca-4801-bc41-2f734374bc8f" (UID: "dd605101-22ca-4801-bc41-2f734374bc8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:01.609471 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.609450 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume" pod "b787ffbb-5157-475b-8eb2-b7c6f5e8671d" (UID: "b787ffbb-5157-475b-8eb2-b7c6f5e8671d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:01.610999 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.610971 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg" pod "b787ffbb-5157-475b-8eb2-b7c6f5e8671d" (UID: "b787ffbb-5157-475b-8eb2-b7c6f5e8671d"). InnerVolumeSpecName "kube-api-access-fnwmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:01.611073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.611014 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f" pod "dd605101-22ca-4801-bc41-2f734374bc8f" (UID: "dd605101-22ca-4801-bc41-2f734374bc8f"). InnerVolumeSpecName "kube-api-access-zn86f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:01.710465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.710427 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd605101-22ca-4801-bc41-2f734374bc8f-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:01.710465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.710457 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn86f\" (UniqueName: \"kubernetes.io/projected/dd605101-22ca-4801-bc41-2f734374bc8f-kube-api-access-zn86f\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:01.710465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.710468 7074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-plugins-config-volume\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:01.710716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:01.710478 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnwmg\" (UniqueName: \"kubernetes.io/projected/b787ffbb-5157-475b-8eb2-b7c6f5e8671d-kube-api-access-fnwmg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:02.160103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.160075 7074 generic.go:356] "Generic (PLEG): container finished" podID="dd605101-22ca-4801-bc41-2f734374bc8f" containerID="3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb" exitCode=0 Oct 31 23:26:02.160210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.160122 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" Oct 31 23:26:02.160210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.160150 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" event={"ID":"dd605101-22ca-4801-bc41-2f734374bc8f","Type":"ContainerDied","Data":"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb"} Oct 31 23:26:02.160210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.160197 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl" event={"ID":"dd605101-22ca-4801-bc41-2f734374bc8f","Type":"ContainerDied","Data":"ffea93880a501204b45dad136ad8714d84201437e8d15e52bf63364ad118972f"} Oct 31 23:26:02.160331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.160220 7074 scope.go:122] "RemoveContainer" containerID="3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb" Oct 31 23:26:02.161513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.161493 7074 generic.go:356] "Generic (PLEG): container finished" podID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerID="b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485" exitCode=0 Oct 31 23:26:02.161630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.161527 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" event={"ID":"b787ffbb-5157-475b-8eb2-b7c6f5e8671d","Type":"ContainerDied","Data":"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485"} Oct 31 23:26:02.161630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.161555 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" event={"ID":"b787ffbb-5157-475b-8eb2-b7c6f5e8671d","Type":"ContainerDied","Data":"4a37bd0bdee735649e843673126d926b7c87716e5fe7a80e9e914a5f3ec0f1b3"} Oct 31 23:26:02.161630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.161529 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq" Oct 31 23:26:02.170468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.170449 7074 scope.go:122] "RemoveContainer" containerID="3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb" Oct 31 23:26:02.170776 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:02.170749 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb\": container with ID starting with 3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb not found: ID does not exist" containerID="3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb" Oct 31 23:26:02.170833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.170787 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb"} err="failed to get container status \"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb\": rpc error: code = NotFound desc = could not find container \"3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb\": container with ID starting with 3eabad8ca45b5af2e70c30deac878629e001e877e77115a90b011981321efcfb not found: ID does not exist" Oct 31 23:26:02.170833 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.170804 7074 scope.go:122] "RemoveContainer" containerID="b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485" Oct 31 23:26:02.179120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.179087 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:26:02.180693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.180674 7074 scope.go:122] "RemoveContainer" containerID="b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485" Oct 31 23:26:02.181120 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:02.181094 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485\": container with ID starting with b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485 not found: ID does not exist" containerID="b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485" Oct 31 23:26:02.181291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.181127 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485"} err="failed to get container status \"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485\": rpc error: code = NotFound desc = could not find container \"b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485\": container with ID starting with b19cd5d215e7b29017e47d64e3e0a05fbefafa9bc405e7c056bd8f4fd3d9b485 not found: ID does not exist" Oct 31 23:26:02.183579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.183557 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-255748-757d5866d8-hjmfq"] Oct 31 23:26:02.195545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.195513 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:26:02.202356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.202281 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-255748-6cd56b79f-zwgwl"] Oct 31 23:26:02.222976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.222843 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:26:02.223531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.223499 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-69b96f8484-7gg8s" podUID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" containerName="manager" containerID="cri-o://1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52" gracePeriod=30 Oct 31 23:26:02.484335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.484306 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:26:02.519192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.518822 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\") pod \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " Oct 31 23:26:02.519192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.518884 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\" (UniqueName: \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\") pod \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " Oct 31 23:26:02.519192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.519092 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\") pod \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\" (UID: \"0a35fd31-2d43-452e-9871-0f4d6b569aeb\") " Oct 31 23:26:02.521913 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.521881 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert" pod "0a35fd31-2d43-452e-9871-0f4d6b569aeb" (UID: "0a35fd31-2d43-452e-9871-0f4d6b569aeb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:26:02.522132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.522058 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs" pod "0a35fd31-2d43-452e-9871-0f4d6b569aeb" (UID: "0a35fd31-2d43-452e-9871-0f4d6b569aeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:26:02.523729 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.523698 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj" pod "0a35fd31-2d43-452e-9871-0f4d6b569aeb" (UID: "0a35fd31-2d43-452e-9871-0f4d6b569aeb"). InnerVolumeSpecName "kube-api-access-w5ltj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:02.620279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.620242 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:02.620279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.620273 7074 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a35fd31-2d43-452e-9871-0f4d6b569aeb-webhook-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:02.620279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:02.620283 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ltj\" (UniqueName: \"kubernetes.io/projected/0a35fd31-2d43-452e-9871-0f4d6b569aeb-kube-api-access-w5ltj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:03.170017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.169956 7074 generic.go:356] "Generic (PLEG): container finished" podID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" containerID="1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52" exitCode=0 Oct 31 23:26:03.170017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.170017 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-7gg8s" Oct 31 23:26:03.170280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.170024 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-7gg8s" event={"ID":"0a35fd31-2d43-452e-9871-0f4d6b569aeb","Type":"ContainerDied","Data":"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52"} Oct 31 23:26:03.170280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.170081 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-7gg8s" event={"ID":"0a35fd31-2d43-452e-9871-0f4d6b569aeb","Type":"ContainerDied","Data":"5b8a953b4ba9c10510d800ce103ae412ee1774b7fd99720cf77455a921dc40ed"} Oct 31 23:26:03.170280 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.170110 7074 scope.go:122] "RemoveContainer" containerID="1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52" Oct 31 23:26:03.180680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.180659 7074 scope.go:122] "RemoveContainer" containerID="1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52" Oct 31 23:26:03.180938 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:03.180918 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52\": container with ID starting with 1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52 not found: ID does not exist" containerID="1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52" Oct 31 23:26:03.181012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.180946 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52"} err="failed to get container status \"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52\": rpc error: code = NotFound desc = could not find container \"1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52\": container with ID starting with 1c902cd318256e1b58874aa02e613e192737250e6c5ab0d66fe86a6be3ad5a52 not found: ID does not exist" Oct 31 23:26:03.192631 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.192608 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:26:03.197023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:03.196984 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69b96f8484-7gg8s"] Oct 31 23:26:04.017386 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017350 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:04.017911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017865 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" Oct 31 23:26:04.017911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017879 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" Oct 31 23:26:04.017911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017900 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" Oct 31 23:26:04.017911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017906 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" Oct 31 23:26:04.017911 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017915 7074 cpu_manager.go:414] "containerMap: removing container" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017920 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017927 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017932 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017943 7074 cpu_manager.go:414] "containerMap: removing container" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017948 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017954 7074 cpu_manager.go:414] "containerMap: removing container" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017959 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017968 7074 cpu_manager.go:414] "containerMap: removing container" podUID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017976 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.017983 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018001 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018010 7074 cpu_manager.go:414] "containerMap: removing container" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018016 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018026 7074 cpu_manager.go:414] "containerMap: removing container" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018030 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018040 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018047 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018057 7074 cpu_manager.go:414] "containerMap: removing container" podUID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018062 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018070 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018075 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018082 7074 cpu_manager.go:414] "containerMap: removing container" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018087 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018093 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018097 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018104 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018108 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018114 7074 cpu_manager.go:414] "containerMap: removing container" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018118 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018132 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018138 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018146 7074 cpu_manager.go:414] "containerMap: removing container" podUID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018150 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" containerName="maas-api" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018157 7074 cpu_manager.go:414] "containerMap: removing container" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018161 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018170 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018175 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018183 7074 cpu_manager.go:414] "containerMap: removing container" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018187 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018193 7074 cpu_manager.go:414] "containerMap: removing container" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018198 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018212 7074 cpu_manager.go:414] "containerMap: removing container" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018217 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018224 7074 cpu_manager.go:414] "containerMap: removing container" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018229 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018236 7074 cpu_manager.go:414] "containerMap: removing container" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018240 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018248 7074 cpu_manager.go:414] "containerMap: removing container" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018252 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018258 7074 cpu_manager.go:414] "containerMap: removing container" podUID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" containerName="payload-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018263 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" containerName="payload-processing" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018269 7074 cpu_manager.go:414] "containerMap: removing container" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018273 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" Oct 31 23:26:04.018244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018283 7074 cpu_manager.go:414] "containerMap: removing container" podUID="592e575f-eeea-4d6e-bf00-8e258d9f7929" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018289 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592e575f-eeea-4d6e-bf00-8e258d9f7929" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018300 7074 cpu_manager.go:414] "containerMap: removing container" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018304 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018313 7074 cpu_manager.go:414] "containerMap: removing container" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018316 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018323 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018327 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018333 7074 cpu_manager.go:414] "containerMap: removing container" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018338 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018345 7074 cpu_manager.go:414] "containerMap: removing container" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018350 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018356 7074 cpu_manager.go:414] "containerMap: removing container" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018363 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018373 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" containerName="manager" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018377 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" containerName="manager" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018386 7074 cpu_manager.go:414] "containerMap: removing container" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018392 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018398 7074 cpu_manager.go:414] "containerMap: removing container" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018403 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018410 7074 cpu_manager.go:414] "containerMap: removing container" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018415 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018420 7074 cpu_manager.go:414] "containerMap: removing container" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018425 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018432 7074 cpu_manager.go:414] "containerMap: removing container" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018438 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018446 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b8db220b-9983-4da1-b152-6bc107121c17" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018453 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8db220b-9983-4da1-b152-6bc107121c17" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018458 7074 cpu_manager.go:414] "containerMap: removing container" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018463 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018470 7074 cpu_manager.go:414] "containerMap: removing container" podUID="6db814a8-8d7b-40cb-b938-788f64abf26a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018474 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db814a8-8d7b-40cb-b938-788f64abf26a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018479 7074 cpu_manager.go:414] "containerMap: removing container" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018484 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018490 7074 cpu_manager.go:414] "containerMap: removing container" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018494 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018501 7074 cpu_manager.go:414] "containerMap: removing container" podUID="e6d42afb-923e-490b-b388-d2a935f8f12a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018505 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d42afb-923e-490b-b388-d2a935f8f12a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018512 7074 cpu_manager.go:414] "containerMap: removing container" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018518 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018530 7074 cpu_manager.go:414] "containerMap: removing container" podUID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018535 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018540 7074 cpu_manager.go:414] "containerMap: removing container" podUID="3fc753cb-a030-47bc-a64d-dee62ff222b0" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018545 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc753cb-a030-47bc-a64d-dee62ff222b0" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018553 7074 cpu_manager.go:414] "containerMap: removing container" podUID="2a8ce167-55ad-47ac-9cee-731114a42433" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018557 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8ce167-55ad-47ac-9cee-731114a42433" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018628 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12de15d-7208-4703-a91d-ef282c36e0fb" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018638 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="46131fa4-7daa-4a14-bda7-dcc36fc6eedb" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018644 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d48f7b7-120b-4452-9c04-121ffc7ef7ef" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018650 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="15558e00-bb89-4cac-9cd4-866a5f89d589" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018655 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4f68aa-fee5-43a3-b3b0-c768f0a3e123" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018661 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="13751078-df8a-4c5d-a011-4bf0edc29cb9" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018668 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="384aa41f-ca9b-4287-b210-856ebe1eab8a" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018675 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="01be9bb2-abc4-402c-84c9-8ea1f9c6d267" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018684 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="105e3e1e-8500-4de6-b605-a5f79adb683a" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018691 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="592e575f-eeea-4d6e-bf00-8e258d9f7929" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018696 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1b6bef-79f3-4d65-98a7-e8932e6afdc1" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018702 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="821e4f82-82fa-47a5-b918-c2e782987f36" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018708 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e082c5b7-66a7-4edc-8dbb-99a2a3cb88f9" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018714 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="722083a0-8d6b-40e7-be1d-011aaa3df44e" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018718 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e23523-c883-45b6-bd79-a5062bf91962" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018724 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a55d87-5ecd-4ffb-a601-701e66281d43" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018729 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" containerName="manager" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018736 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a779def2-64c5-4553-b53a-0ff1564605ab" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018740 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3248837-588d-4f99-8122-5517412e2120" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018746 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e0bf96-8ee5-4334-a767-f69061825885" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018752 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfea965-40cb-411b-9aa4-4ca947983ed2" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018760 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc3108-a881-484a-bda6-ccc8f9374bbe" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018768 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8db220b-9983-4da1-b152-6bc107121c17" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018774 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a5197a-1b71-42ea-ae3d-705d0396704a" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018780 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db814a8-8d7b-40cb-b938-788f64abf26a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018785 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4421fe12-f738-4e07-9463-a15933118db0" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018790 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0ac0ce-89d2-4837-a771-ba4636017709" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018794 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d42afb-923e-490b-b388-d2a935f8f12a" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018799 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="31668b3b-cfeb-4832-a16d-3ac77c8d6694" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018806 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="374a01e6-c99b-48f0-8fcc-466bf1ea96cb" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018810 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc753cb-a030-47bc-a64d-dee62ff222b0" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018816 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8ce167-55ad-47ac-9cee-731114a42433" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018821 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00fd71e-9c7a-4b34-8bd2-cad937a6ab04" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018828 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018834 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45f0f22-b29b-470e-8e69-57e1298605e6" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018843 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54681c2-f396-49eb-a8a2-5c2fd00f3cac" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018851 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c96086e-8783-457f-9aa5-dd76960b42b5" containerName="payload-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018856 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64337a1-7f1d-4fce-ab33-79092a707849" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018860 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4183fdfd-22de-4bdb-9b13-f3bdc1bbe314" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018866 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" containerName="payload-pre-processing" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018872 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac96af3-4543-4010-a8c3-616ac77264da" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018877 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c7f385-bc1e-43a1-8f3a-b0d7a44dba85" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018882 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c139df47-d3de-4818-805f-05810cdccb12" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018887 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="89021da0-1ab9-4a7c-bda0-4b465e3b801b" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018891 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c7005a-e66f-472e-8dfe-174ff1b66690" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018896 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea39a53-3824-4c95-bb01-edf9a0383e67" containerName="main" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018901 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0825b221-898d-4dcc-bdef-fa0f8673a2c8" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018906 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8285b23-0cb3-409c-8bcd-e933d1e359e1" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018912 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f5523b-cb46-4d56-a70d-c8460db757d9" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018922 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6662d9f-823a-4d16-bf62-86dc28511ef7" containerName="istio-proxy" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018929 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9352f587-17c5-4cb8-a7b3-0e5a517e882f" containerName="maas-api" Oct 31 23:26:04.021224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.018935 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9a0210-fc3a-4c07-a5b0-1b15f2acc656" containerName="istio-proxy" Oct 31 23:26:04.026599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.023620 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.036948 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.035626 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 23:26:04.036948 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.036308 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 23:26:04.040684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.037178 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 23:26:04.040684 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.037632 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vxzrb\"" Oct 31 23:26:04.044807 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.040693 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:04.143560 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.143513 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a35fd31-2d43-452e-9871-0f4d6b569aeb" path="/var/lib/kubelet/pods/0a35fd31-2d43-452e-9871-0f4d6b569aeb/volumes" Oct 31 23:26:04.144175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.144151 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b787ffbb-5157-475b-8eb2-b7c6f5e8671d" path="/var/lib/kubelet/pods/b787ffbb-5157-475b-8eb2-b7c6f5e8671d/volumes" Oct 31 23:26:04.144696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.144674 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd605101-22ca-4801-bc41-2f734374bc8f" path="/var/lib/kubelet/pods/dd605101-22ca-4801-bc41-2f734374bc8f/volumes" Oct 31 23:26:04.158186 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.158150 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4strv\" (UniqueName: \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.158322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.158298 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.158478 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.158454 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.263393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.262676 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.263393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.262784 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4strv\" (UniqueName: \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.263393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.262880 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.282670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.282570 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.283515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.283464 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4strv\" (UniqueName: \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.316016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.315766 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\") pod \"maas-controller-69b96f8484-2b76g\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.362383 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.362351 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:04.809015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:04.806015 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:05.087065 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.073210 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:26:05.224221 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.224180 7074 generic.go:356] "Generic (PLEG): container finished" podID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerID="10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c" exitCode=0 Oct 31 23:26:05.224577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.224229 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" Oct 31 23:26:05.224805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.224244 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" event={"ID":"c193e2ae-7f72-4030-9cf9-d930c8c8255a","Type":"ContainerDied","Data":"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c"} Oct 31 23:26:05.225005 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.224956 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n" event={"ID":"c193e2ae-7f72-4030-9cf9-d930c8c8255a","Type":"ContainerDied","Data":"d7b0aa1e1bc6d443237169db4fd8a5c1449e1a9734b98738150e4fb49c0cc078"} Oct 31 23:26:05.225119 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225025 7074 scope.go:122] "RemoveContainer" containerID="10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c" Oct 31 23:26:05.225413 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225389 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225456 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225557 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225507 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225624 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225597 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225662 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225685 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.225753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225719 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225748 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225822 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225860 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\") pod \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\" (UID: \"c193e2ae-7f72-4030-9cf9-d930c8c8255a\") " Oct 31 23:26:05.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225865 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.225962 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.225914 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.226255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.226215 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.226255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.226228 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.226255 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.226239 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.226414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.226371 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.226414 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.226301 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:05.227190 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.227155 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-2b76g" event={"ID":"2d0f6372-6926-41d9-b97a-368c16c8e91e","Type":"ContainerStarted","Data":"07eebd62048fcc5071ead09babb397e1dbd609ac2c79b5f3e17c855b128f0af5"} Oct 31 23:26:05.228277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.228256 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.228391 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.228371 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:26:05.228552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.228536 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:05.228605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.228587 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs" pod "c193e2ae-7f72-4030-9cf9-d930c8c8255a" (UID: "c193e2ae-7f72-4030-9cf9-d930c8c8255a"). InnerVolumeSpecName "kube-api-access-b5hzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:05.235674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.235606 7074 scope.go:122] "RemoveContainer" containerID="10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c" Oct 31 23:26:05.235976 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:05.235922 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c\": container with ID starting with 10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c not found: ID does not exist" containerID="10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c" Oct 31 23:26:05.235976 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.235952 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c"} err="failed to get container status \"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c\": rpc error: code = NotFound desc = could not find container \"10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c\": container with ID starting with 10972cbf8af23e8f78b6ed4c29ed52e5cd4d1f42e366a6654ff7809cd807b41c not found: ID does not exist" Oct 31 23:26:05.328791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328756 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hzs\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-kube-api-access-b5hzs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.328908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328798 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.328908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328816 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.328908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328831 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.328908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328844 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.328908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.328858 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193e2ae-7f72-4030-9cf9-d930c8c8255a-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:05.593371 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.592425 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:26:05.614194 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:05.606043 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-w7z8n"] Oct 31 23:26:06.141649 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:06.141616 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" path="/var/lib/kubelet/pods/c193e2ae-7f72-4030-9cf9-d930c8c8255a/volumes" Oct 31 23:26:06.234420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:06.234384 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-2b76g" event={"ID":"2d0f6372-6926-41d9-b97a-368c16c8e91e","Type":"ContainerStarted","Data":"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df"} Oct 31 23:26:06.234604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:06.234443 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:06.298937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:06.298883 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69b96f8484-2b76g" podStartSLOduration=2.663220697 podStartE2EDuration="3.298867013s" totalImagesPullingTime="635.646316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:26:06.297648504 +0000 UTC m=+2144.280390881" watchObservedRunningTime="2026-10-31 23:26:06.298867013 +0000 UTC m=+2144.281609391" Oct 31 23:26:14.887280 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:14.887240 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:26:17.243751 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:17.243721 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:22.085265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.085230 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp"] Oct 31 23:26:22.085913 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.085891 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" Oct 31 23:26:22.086049 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.085915 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" Oct 31 23:26:22.086118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.086088 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c193e2ae-7f72-4030-9cf9-d930c8c8255a" containerName="istio-proxy" Oct 31 23:26:22.089354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.089331 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.097815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.097650 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zpm\" (UniqueName: \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.097815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.097731 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.097815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.097765 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.098908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.098882 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp"] Oct 31 23:26:22.099149 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.099063 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:26:22.099225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.099159 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:26:22.198868 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.198832 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zpm\" (UniqueName: \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.199061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.198875 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.199061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.198911 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.199481 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.199462 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.201849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.201831 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:26:22.211263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.211236 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.211385 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.211373 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zpm\" (UniqueName: \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\") pod \"maas-api-revoke-sub-85237e45-6mxdp\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.411486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.411407 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:26:22.418419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.418394 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:22.695605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:22.695567 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp"] Oct 31 23:26:22.697793 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:26:22.697756 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223a4404_15dd_4a6d_970d_31f3f05f4ff0.slice/crio-5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4 WatchSource:0}: Error finding container 5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4: Status 404 returned error can't find the container with id 5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4 Oct 31 23:26:23.335747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:23.335706 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" event={"ID":"223a4404-15dd-4a6d-970d-31f3f05f4ff0","Type":"ContainerStarted","Data":"15ec22812464cd8a6b0e5cf1541cd451f92b42318941cdea59b2a656f9f0fa9f"} Oct 31 23:26:23.336204 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:23.335786 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" event={"ID":"223a4404-15dd-4a6d-970d-31f3f05f4ff0","Type":"ContainerStarted","Data":"5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4"} Oct 31 23:26:23.356036 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:23.355973 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" podStartSLOduration=1.355959942 podStartE2EDuration="1.355959942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:26:23.353761884 +0000 UTC m=+2161.336504257" watchObservedRunningTime="2026-10-31 23:26:23.355959942 +0000 UTC m=+2161.338702320" Oct 31 23:26:24.343801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:24.343762 7074 generic.go:356] "Generic (PLEG): container finished" podID="223a4404-15dd-4a6d-970d-31f3f05f4ff0" containerID="15ec22812464cd8a6b0e5cf1541cd451f92b42318941cdea59b2a656f9f0fa9f" exitCode=0 Oct 31 23:26:24.344214 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:24.343837 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" event={"ID":"223a4404-15dd-4a6d-970d-31f3f05f4ff0","Type":"ContainerDied","Data":"15ec22812464cd8a6b0e5cf1541cd451f92b42318941cdea59b2a656f9f0fa9f"} Oct 31 23:26:25.483784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.483756 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:25.526971 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.526942 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\" (UniqueName: \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\") pod \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " Oct 31 23:26:25.527150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.527073 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\") pod \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " Oct 31 23:26:25.529025 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.528973 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm" pod "223a4404-15dd-4a6d-970d-31f3f05f4ff0" (UID: "223a4404-15dd-4a6d-970d-31f3f05f4ff0"). InnerVolumeSpecName "kube-api-access-n6zpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:25.529132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.529091 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca" pod "223a4404-15dd-4a6d-970d-31f3f05f4ff0" (UID: "223a4404-15dd-4a6d-970d-31f3f05f4ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:26:25.628259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.628188 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\") pod \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\" (UID: \"223a4404-15dd-4a6d-970d-31f3f05f4ff0\") " Oct 31 23:26:25.628498 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.628485 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:25.628545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.628492 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca" pod "223a4404-15dd-4a6d-970d-31f3f05f4ff0" (UID: "223a4404-15dd-4a6d-970d-31f3f05f4ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:25.628545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.628500 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zpm\" (UniqueName: \"kubernetes.io/projected/223a4404-15dd-4a6d-970d-31f3f05f4ff0-kube-api-access-n6zpm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:25.729084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:25.729049 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223a4404-15dd-4a6d-970d-31f3f05f4ff0-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:26.355927 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:26.355893 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" Oct 31 23:26:26.356140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:26.355949 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp" event={"ID":"223a4404-15dd-4a6d-970d-31f3f05f4ff0","Type":"ContainerDied","Data":"5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4"} Oct 31 23:26:26.356140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:26.356020 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c90919d13a2718d0ce2255e7c540cc07603f923337d9124bb6eb198bda1a6c4" Oct 31 23:26:33.718824 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:33.718725 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:33.719648 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:33.719585 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-69b96f8484-2b76g" podUID="2d0f6372-6926-41d9-b97a-368c16c8e91e" containerName="manager" containerID="cri-o://825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df" gracePeriod=30 Oct 31 23:26:33.971613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:33.971552 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:34.003450 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.003422 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\") pod \"2d0f6372-6926-41d9-b97a-368c16c8e91e\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " Oct 31 23:26:34.003658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.003469 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\") pod \"2d0f6372-6926-41d9-b97a-368c16c8e91e\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " Oct 31 23:26:34.003658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.003590 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\" (UniqueName: \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\") pod \"2d0f6372-6926-41d9-b97a-368c16c8e91e\" (UID: \"2d0f6372-6926-41d9-b97a-368c16c8e91e\") " Oct 31 23:26:34.006193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.006161 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs" pod "2d0f6372-6926-41d9-b97a-368c16c8e91e" (UID: "2d0f6372-6926-41d9-b97a-368c16c8e91e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:26:34.008797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.007118 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv" pod "2d0f6372-6926-41d9-b97a-368c16c8e91e" (UID: "2d0f6372-6926-41d9-b97a-368c16c8e91e"). InnerVolumeSpecName "kube-api-access-4strv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:34.008797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.007344 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert" pod "2d0f6372-6926-41d9-b97a-368c16c8e91e" (UID: "2d0f6372-6926-41d9-b97a-368c16c8e91e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:26:34.105533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.105502 7074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-metrics-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:34.105533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.105532 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4strv\" (UniqueName: \"kubernetes.io/projected/2d0f6372-6926-41d9-b97a-368c16c8e91e-kube-api-access-4strv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:34.105726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.105543 7074 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d0f6372-6926-41d9-b97a-368c16c8e91e-webhook-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:34.405003 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.404967 7074 generic.go:356] "Generic (PLEG): container finished" podID="2d0f6372-6926-41d9-b97a-368c16c8e91e" containerID="825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df" exitCode=0 Oct 31 23:26:34.405220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.405044 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-2b76g" Oct 31 23:26:34.405220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.405058 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-2b76g" event={"ID":"2d0f6372-6926-41d9-b97a-368c16c8e91e","Type":"ContainerDied","Data":"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df"} Oct 31 23:26:34.405220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.405152 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-2b76g" event={"ID":"2d0f6372-6926-41d9-b97a-368c16c8e91e","Type":"ContainerDied","Data":"07eebd62048fcc5071ead09babb397e1dbd609ac2c79b5f3e17c855b128f0af5"} Oct 31 23:26:34.405220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.405182 7074 scope.go:122] "RemoveContainer" containerID="825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df" Oct 31 23:26:34.414551 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.414532 7074 scope.go:122] "RemoveContainer" containerID="825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df" Oct 31 23:26:34.414846 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:34.414826 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df\": container with ID starting with 825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df not found: ID does not exist" containerID="825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df" Oct 31 23:26:34.414909 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.414858 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df"} err="failed to get container status \"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df\": rpc error: code = NotFound desc = could not find container \"825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df\": container with ID starting with 825eb9bc1a7ae4f161ffc48b8bf407d3a0f6303da1b19c987dd1ad74c7b8d6df not found: ID does not exist" Oct 31 23:26:34.422894 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.422872 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:34.426950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:34.426920 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69b96f8484-2b76g"] Oct 31 23:26:35.679558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.679518 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69b96f8484-9965q"] Oct 31 23:26:35.680046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680036 7074 cpu_manager.go:414] "containerMap: removing container" podUID="2d0f6372-6926-41d9-b97a-368c16c8e91e" containerName="manager" Oct 31 23:26:35.680101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680051 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0f6372-6926-41d9-b97a-368c16c8e91e" containerName="manager" Oct 31 23:26:35.680101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680062 7074 cpu_manager.go:414] "containerMap: removing container" podUID="223a4404-15dd-4a6d-970d-31f3f05f4ff0" containerName="revoke-keys" Oct 31 23:26:35.680101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680067 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223a4404-15dd-4a6d-970d-31f3f05f4ff0" containerName="revoke-keys" Oct 31 23:26:35.680202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680152 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0f6372-6926-41d9-b97a-368c16c8e91e" containerName="manager" Oct 31 23:26:35.680202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.680160 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="223a4404-15dd-4a6d-970d-31f3f05f4ff0" containerName="revoke-keys" Oct 31 23:26:35.684617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.684601 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.688888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.688866 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 23:26:35.690205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.690182 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 23:26:35.690205 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.690182 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vxzrb\"" Oct 31 23:26:35.690381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.690251 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 23:26:35.695137 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.695111 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-9965q"] Oct 31 23:26:35.722418 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.722376 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-metrics-certs\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.722597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.722522 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhv5\" (UniqueName: \"kubernetes.io/projected/0442774d-6d62-408e-b222-8f5e57490583-kube-api-access-vrhv5\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.722644 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.722620 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-webhook-cert\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.823957 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.823913 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhv5\" (UniqueName: \"kubernetes.io/projected/0442774d-6d62-408e-b222-8f5e57490583-kube-api-access-vrhv5\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.824182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.823981 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-webhook-cert\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.824182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.824058 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-metrics-certs\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.826486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.826456 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-metrics-certs\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.826598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.826497 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0442774d-6d62-408e-b222-8f5e57490583-webhook-cert\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.836287 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.836265 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhv5\" (UniqueName: \"kubernetes.io/projected/0442774d-6d62-408e-b222-8f5e57490583-kube-api-access-vrhv5\") pod \"maas-controller-69b96f8484-9965q\" (UID: \"0442774d-6d62-408e-b222-8f5e57490583\") " pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:35.998786 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:35.998756 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:36.141793 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:36.141759 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0f6372-6926-41d9-b97a-368c16c8e91e" path="/var/lib/kubelet/pods/2d0f6372-6926-41d9-b97a-368c16c8e91e/volumes" Oct 31 23:26:36.282440 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:36.282413 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69b96f8484-9965q"] Oct 31 23:26:36.284554 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:26:36.284528 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0442774d_6d62_408e_b222_8f5e57490583.slice/crio-f1f2a9af138c2361b657f7d71bea1fe2a773ba5874e5d6097e5e3f9bd5ffe966 WatchSource:0}: Error finding container f1f2a9af138c2361b657f7d71bea1fe2a773ba5874e5d6097e5e3f9bd5ffe966: Status 404 returned error can't find the container with id f1f2a9af138c2361b657f7d71bea1fe2a773ba5874e5d6097e5e3f9bd5ffe966 Oct 31 23:26:36.419336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:36.419299 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-9965q" event={"ID":"0442774d-6d62-408e-b222-8f5e57490583","Type":"ContainerStarted","Data":"f1f2a9af138c2361b657f7d71bea1fe2a773ba5874e5d6097e5e3f9bd5ffe966"} Oct 31 23:26:37.426919 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:37.426887 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69b96f8484-9965q" event={"ID":"0442774d-6d62-408e-b222-8f5e57490583","Type":"ContainerStarted","Data":"30ae740d5ccf199d55a8386cc6356783108e6c500b0d909f62ccb2b6d3da1622"} Oct 31 23:26:37.427333 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:37.427019 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:26:37.447791 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:37.447746 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69b96f8484-9965q" podStartSLOduration=2.029974727 podStartE2EDuration="2.447730755s" totalImagesPullingTime="417.756028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:26:37.446916871 +0000 UTC m=+2175.429659248" watchObservedRunningTime="2026-10-31 23:26:37.447730755 +0000 UTC m=+2175.430473137" Oct 31 23:26:44.888611 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:26:44.888564 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:26:48.435279 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:26:48.435248 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69b96f8484-9965q" Oct 31 23:27:14.891679 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:27:14.891619 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:27:15.584225 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.584191 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh"] Oct 31 23:27:15.588248 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.588231 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.596148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.596115 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:27:15.596148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.596115 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:27:15.599031 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.599001 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh"] Oct 31 23:27:15.665167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.665130 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmv9w\" (UniqueName: \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.665336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.665279 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.665336 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.665316 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.766965 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.766931 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.767151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.766977 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.767151 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.767055 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmv9w\" (UniqueName: \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.767767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.767742 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.769176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.769159 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.780464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.780443 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmv9w\" (UniqueName: \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\") pod \"maas-api-revoke-sub-c1e271a9-prmvh\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:15.903721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:15.903634 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:16.181517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:16.181490 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh"] Oct 31 23:27:16.183695 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:27:16.183662 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d83fb75_70a2_4329_b759_4ece9c4ce55d.slice/crio-195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7 WatchSource:0}: Error finding container 195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7: Status 404 returned error can't find the container with id 195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7 Oct 31 23:27:16.657801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:16.657758 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" event={"ID":"1d83fb75-70a2-4329-b759-4ece9c4ce55d","Type":"ContainerStarted","Data":"1b3358e5fe43e7c27b6267f12c4f865c385c40b325e43a8bc2ce0bff751195cf"} Oct 31 23:27:16.657980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:16.657859 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" event={"ID":"1d83fb75-70a2-4329-b759-4ece9c4ce55d","Type":"ContainerStarted","Data":"195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7"} Oct 31 23:27:16.682095 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:16.682042 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" podStartSLOduration=1.682028607 podStartE2EDuration="1.682028607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:27:16.680343492 +0000 UTC m=+2214.663085870" watchObservedRunningTime="2026-10-31 23:27:16.682028607 +0000 UTC m=+2214.664770985" Oct 31 23:27:17.664739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:17.664698 7074 generic.go:356] "Generic (PLEG): container finished" podID="1d83fb75-70a2-4329-b759-4ece9c4ce55d" containerID="1b3358e5fe43e7c27b6267f12c4f865c385c40b325e43a8bc2ce0bff751195cf" exitCode=0 Oct 31 23:27:17.665220 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:17.664782 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" event={"ID":"1d83fb75-70a2-4329-b759-4ece9c4ce55d","Type":"ContainerDied","Data":"1b3358e5fe43e7c27b6267f12c4f865c385c40b325e43a8bc2ce0bff751195cf"} Oct 31 23:27:18.812585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.812559 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:18.896865 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.896836 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\") pod \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " Oct 31 23:27:18.897104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.896897 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\" (UniqueName: \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\") pod \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " Oct 31 23:27:18.897104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.896930 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\") pod \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\" (UID: \"1d83fb75-70a2-4329-b759-4ece9c4ce55d\") " Oct 31 23:27:18.897321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.897298 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca" pod "1d83fb75-70a2-4329-b759-4ece9c4ce55d" (UID: "1d83fb75-70a2-4329-b759-4ece9c4ce55d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:27:18.899147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.899121 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w" pod "1d83fb75-70a2-4329-b759-4ece9c4ce55d" (UID: "1d83fb75-70a2-4329-b759-4ece9c4ce55d"). InnerVolumeSpecName "kube-api-access-bmv9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:27:18.899147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.899139 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca" pod "1d83fb75-70a2-4329-b759-4ece9c4ce55d" (UID: "1d83fb75-70a2-4329-b759-4ece9c4ce55d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:27:18.998598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.998559 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:18.998598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.998595 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmv9w\" (UniqueName: \"kubernetes.io/projected/1d83fb75-70a2-4329-b759-4ece9c4ce55d-kube-api-access-bmv9w\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:18.998795 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:18.998606 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d83fb75-70a2-4329-b759-4ece9c4ce55d-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:19.676908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:19.676875 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" event={"ID":"1d83fb75-70a2-4329-b759-4ece9c4ce55d","Type":"ContainerDied","Data":"195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7"} Oct 31 23:27:19.677114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:19.676912 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh" Oct 31 23:27:19.677114 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:19.676925 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195ba90f8b67da6cc03eb7afd64daa17afca45d0b83a8973e5b7b6484f7da6f7" Oct 31 23:27:34.514024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.513968 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm"] Oct 31 23:27:34.514553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.514535 7074 cpu_manager.go:414] "containerMap: removing container" podUID="1d83fb75-70a2-4329-b759-4ece9c4ce55d" containerName="revoke-keys" Oct 31 23:27:34.514603 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.514556 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d83fb75-70a2-4329-b759-4ece9c4ce55d" containerName="revoke-keys" Oct 31 23:27:34.514682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.514673 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d83fb75-70a2-4329-b759-4ece9c4ce55d" containerName="revoke-keys" Oct 31 23:27:34.519259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.519242 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.527849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.526850 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:27:34.527849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.527774 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:27:34.532502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.532473 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm"] Oct 31 23:27:34.573312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.573280 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.573484 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.573326 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc9pm\" (UniqueName: \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.573559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.573528 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.675156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.675119 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.675156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.675160 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc9pm\" (UniqueName: \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.675419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.675229 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.675822 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.675799 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.677587 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.677561 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.687697 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.687673 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc9pm\" (UniqueName: \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\") pod \"maas-api-revoke-sub-2e1f5b5f-59smm\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:34.848780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:34.848695 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:35.123461 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:35.123437 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm"] Oct 31 23:27:35.125977 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:27:35.125936 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df9042d_2469_4337_9c47_ebf03774d693.slice/crio-fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2 WatchSource:0}: Error finding container fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2: Status 404 returned error can't find the container with id fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2 Oct 31 23:27:35.811731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:35.811696 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" event={"ID":"1df9042d-2469-4337-9c47-ebf03774d693","Type":"ContainerStarted","Data":"0bd0351a78db73a94275610cc64f2d3afda7a9a2bde90ae9f465be6ce2cf9c69"} Oct 31 23:27:35.812140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:35.811752 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" event={"ID":"1df9042d-2469-4337-9c47-ebf03774d693","Type":"ContainerStarted","Data":"fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2"} Oct 31 23:27:35.831601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:35.831541 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" podStartSLOduration=1.831526465 podStartE2EDuration="1.831526465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:27:35.829669726 +0000 UTC m=+2233.812412105" watchObservedRunningTime="2026-10-31 23:27:35.831526465 +0000 UTC m=+2233.814268880" Oct 31 23:27:36.820668 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:36.820632 7074 generic.go:356] "Generic (PLEG): container finished" podID="1df9042d-2469-4337-9c47-ebf03774d693" containerID="0bd0351a78db73a94275610cc64f2d3afda7a9a2bde90ae9f465be6ce2cf9c69" exitCode=0 Oct 31 23:27:36.821136 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:36.820712 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" event={"ID":"1df9042d-2469-4337-9c47-ebf03774d693","Type":"ContainerDied","Data":"0bd0351a78db73a94275610cc64f2d3afda7a9a2bde90ae9f465be6ce2cf9c69"} Oct 31 23:27:37.967146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:37.967124 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:38.109437 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.109359 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\") pod \"1df9042d-2469-4337-9c47-ebf03774d693\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " Oct 31 23:27:38.109586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.109455 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\" (UniqueName: \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\") pod \"1df9042d-2469-4337-9c47-ebf03774d693\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " Oct 31 23:27:38.109586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.109504 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\") pod \"1df9042d-2469-4337-9c47-ebf03774d693\" (UID: \"1df9042d-2469-4337-9c47-ebf03774d693\") " Oct 31 23:27:38.109714 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.109692 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca" pod "1df9042d-2469-4337-9c47-ebf03774d693" (UID: "1df9042d-2469-4337-9c47-ebf03774d693"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:27:38.109891 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.109868 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:38.111506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.111486 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm" pod "1df9042d-2469-4337-9c47-ebf03774d693" (UID: "1df9042d-2469-4337-9c47-ebf03774d693"). InnerVolumeSpecName "kube-api-access-xc9pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:27:38.111593 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.111577 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca" pod "1df9042d-2469-4337-9c47-ebf03774d693" (UID: "1df9042d-2469-4337-9c47-ebf03774d693"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:27:38.210455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.210417 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc9pm\" (UniqueName: \"kubernetes.io/projected/1df9042d-2469-4337-9c47-ebf03774d693-kube-api-access-xc9pm\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:38.210455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.210449 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df9042d-2469-4337-9c47-ebf03774d693-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:38.831857 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.831822 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" Oct 31 23:27:38.832032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.831854 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm" event={"ID":"1df9042d-2469-4337-9c47-ebf03774d693","Type":"ContainerDied","Data":"fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2"} Oct 31 23:27:38.832032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:38.831895 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba74c4dd1e2c0a8e6fd1fca3a2b02ba4fc3972e80cd317cafac20e2880433c2" Oct 31 23:27:44.889501 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:27:44.889459 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:27:45.588242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.588208 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88"] Oct 31 23:27:45.588770 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.588751 7074 cpu_manager.go:414] "containerMap: removing container" podUID="1df9042d-2469-4337-9c47-ebf03774d693" containerName="revoke-keys" Oct 31 23:27:45.588873 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.588772 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df9042d-2469-4337-9c47-ebf03774d693" containerName="revoke-keys" Oct 31 23:27:45.588952 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.588939 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df9042d-2469-4337-9c47-ebf03774d693" containerName="revoke-keys" Oct 31 23:27:45.592389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.592361 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.602032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.598697 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:27:45.602032 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.600571 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:27:45.604428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.604401 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88"] Oct 31 23:27:45.687559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.687517 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfxp\" (UniqueName: \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.687559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.687563 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.687801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.687586 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.788713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.788670 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.788906 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.788724 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.789485 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.789001 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfxp\" (UniqueName: \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.789939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.789705 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.792564 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.792536 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.801667 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.801639 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfxp\" (UniqueName: \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\") pod \"maas-api-revoke-sub-ec843018-scx88\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:45.909261 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:45.909179 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:46.186042 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:46.186014 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88"] Oct 31 23:27:46.189034 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:27:46.189004 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255046e8_664c_4cb2_b2da_c3b70e7126eb.slice/crio-566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580 WatchSource:0}: Error finding container 566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580: Status 404 returned error can't find the container with id 566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580 Oct 31 23:27:46.876669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:46.876630 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" event={"ID":"255046e8-664c-4cb2-b2da-c3b70e7126eb","Type":"ContainerStarted","Data":"bad5e87f4205b832cd1b93531f44c8bacd65725ccc2ff738b6cca9821426b8f8"} Oct 31 23:27:46.876810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:46.876712 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" event={"ID":"255046e8-664c-4cb2-b2da-c3b70e7126eb","Type":"ContainerStarted","Data":"566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580"} Oct 31 23:27:46.898547 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:46.898497 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" podStartSLOduration=1.8984839359999999 podStartE2EDuration="1.898483936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:27:46.897561533 +0000 UTC m=+2244.880303910" watchObservedRunningTime="2026-10-31 23:27:46.898483936 +0000 UTC m=+2244.881226342" Oct 31 23:27:47.886424 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:47.886386 7074 generic.go:356] "Generic (PLEG): container finished" podID="255046e8-664c-4cb2-b2da-c3b70e7126eb" containerID="bad5e87f4205b832cd1b93531f44c8bacd65725ccc2ff738b6cca9821426b8f8" exitCode=0 Oct 31 23:27:47.886846 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:47.886487 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" event={"ID":"255046e8-664c-4cb2-b2da-c3b70e7126eb","Type":"ContainerDied","Data":"bad5e87f4205b832cd1b93531f44c8bacd65725ccc2ff738b6cca9821426b8f8"} Oct 31 23:27:49.323037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.322974 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:49.436781 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.436743 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\" (UniqueName: \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\") pod \"255046e8-664c-4cb2-b2da-c3b70e7126eb\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " Oct 31 23:27:49.436974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.436840 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\") pod \"255046e8-664c-4cb2-b2da-c3b70e7126eb\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " Oct 31 23:27:49.437110 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.437055 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\") pod \"255046e8-664c-4cb2-b2da-c3b70e7126eb\" (UID: \"255046e8-664c-4cb2-b2da-c3b70e7126eb\") " Oct 31 23:27:49.437464 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.437439 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca" pod "255046e8-664c-4cb2-b2da-c3b70e7126eb" (UID: "255046e8-664c-4cb2-b2da-c3b70e7126eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:27:49.439101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.439079 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca" pod "255046e8-664c-4cb2-b2da-c3b70e7126eb" (UID: "255046e8-664c-4cb2-b2da-c3b70e7126eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:27:49.439196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.439144 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp" pod "255046e8-664c-4cb2-b2da-c3b70e7126eb" (UID: "255046e8-664c-4cb2-b2da-c3b70e7126eb"). InnerVolumeSpecName "kube-api-access-5cfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:27:49.538681 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.538643 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:49.538681 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.538674 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cfxp\" (UniqueName: \"kubernetes.io/projected/255046e8-664c-4cb2-b2da-c3b70e7126eb-kube-api-access-5cfxp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:49.538681 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.538685 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/255046e8-664c-4cb2-b2da-c3b70e7126eb-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:27:49.917330 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.917241 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" event={"ID":"255046e8-664c-4cb2-b2da-c3b70e7126eb","Type":"ContainerDied","Data":"566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580"} Oct 31 23:27:49.917646 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.917280 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88" Oct 31 23:27:49.917805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:27:49.917531 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="566be01c354474f952fb91c7b8ad05f3e2abf43e1cfbe522b2db68e7f54c1580" Oct 31 23:28:12.175829 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.175744 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7"] Oct 31 23:28:12.176516 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.176495 7074 cpu_manager.go:414] "containerMap: removing container" podUID="255046e8-664c-4cb2-b2da-c3b70e7126eb" containerName="revoke-keys" Oct 31 23:28:12.176615 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.176518 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255046e8-664c-4cb2-b2da-c3b70e7126eb" containerName="revoke-keys" Oct 31 23:28:12.176671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.176621 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="255046e8-664c-4cb2-b2da-c3b70e7126eb" containerName="revoke-keys" Oct 31 23:28:12.180341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.180314 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.187572 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.187546 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:28:12.188164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.187563 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:28:12.191583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.191558 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7"] Oct 31 23:28:12.225683 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.225651 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.225855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.225691 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfjz\" (UniqueName: \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.225855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.225845 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.326864 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.326829 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.327106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.326890 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.327106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.326912 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfjz\" (UniqueName: \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.327589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.327567 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.329352 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.329329 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.339629 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.339600 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfjz\" (UniqueName: \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\") pod \"maas-api-revoke-sub-046e1343-7s9j7\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.496310 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.496224 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:12.770869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:12.770843 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7"] Oct 31 23:28:12.772848 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:28:12.772814 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0479fde5_6457_44ad_a1ab_40ad945c563d.slice/crio-fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a WatchSource:0}: Error finding container fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a: Status 404 returned error can't find the container with id fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a Oct 31 23:28:13.076623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:13.076587 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" event={"ID":"0479fde5-6457-44ad-a1ab-40ad945c563d","Type":"ContainerStarted","Data":"ea8beee6dae8c89bcc0b9b1846ed4a198dc0eddd665fd8d84381bde6503824bd"} Oct 31 23:28:13.076771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:13.076649 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" event={"ID":"0479fde5-6457-44ad-a1ab-40ad945c563d","Type":"ContainerStarted","Data":"fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a"} Oct 31 23:28:13.096112 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:13.096065 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" podStartSLOduration=1.096051024 podStartE2EDuration="1.096051024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:28:13.094416967 +0000 UTC m=+2271.077159344" watchObservedRunningTime="2026-10-31 23:28:13.096051024 +0000 UTC m=+2271.078793402" Oct 31 23:28:14.084076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:14.084041 7074 generic.go:356] "Generic (PLEG): container finished" podID="0479fde5-6457-44ad-a1ab-40ad945c563d" containerID="ea8beee6dae8c89bcc0b9b1846ed4a198dc0eddd665fd8d84381bde6503824bd" exitCode=0 Oct 31 23:28:14.084480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:14.084128 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" event={"ID":"0479fde5-6457-44ad-a1ab-40ad945c563d","Type":"ContainerDied","Data":"ea8beee6dae8c89bcc0b9b1846ed4a198dc0eddd665fd8d84381bde6503824bd"} Oct 31 23:28:14.887877 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:28:14.887842 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:28:15.233290 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.233265 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:15.358030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.357974 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\") pod \"0479fde5-6457-44ad-a1ab-40ad945c563d\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " Oct 31 23:28:15.358030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.358034 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\") pod \"0479fde5-6457-44ad-a1ab-40ad945c563d\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " Oct 31 23:28:15.358241 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.358093 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\" (UniqueName: \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\") pod \"0479fde5-6457-44ad-a1ab-40ad945c563d\" (UID: \"0479fde5-6457-44ad-a1ab-40ad945c563d\") " Oct 31 23:28:15.358419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.358390 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca" pod "0479fde5-6457-44ad-a1ab-40ad945c563d" (UID: "0479fde5-6457-44ad-a1ab-40ad945c563d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:28:15.360201 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.360176 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca" pod "0479fde5-6457-44ad-a1ab-40ad945c563d" (UID: "0479fde5-6457-44ad-a1ab-40ad945c563d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:28:15.360467 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.360451 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz" pod "0479fde5-6457-44ad-a1ab-40ad945c563d" (UID: "0479fde5-6457-44ad-a1ab-40ad945c563d"). InnerVolumeSpecName "kube-api-access-mdfjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:28:15.459476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.459399 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:15.459476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.459430 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0479fde5-6457-44ad-a1ab-40ad945c563d-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:15.459476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:15.459441 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfjz\" (UniqueName: \"kubernetes.io/projected/0479fde5-6457-44ad-a1ab-40ad945c563d-kube-api-access-mdfjz\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:16.096368 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:16.096332 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" event={"ID":"0479fde5-6457-44ad-a1ab-40ad945c563d","Type":"ContainerDied","Data":"fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a"} Oct 31 23:28:16.096368 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:16.096355 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7" Oct 31 23:28:16.096605 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:16.096385 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf01e7e62244a1e8eed714a1fd5afa9ba5a73005797227c14c896e1d34b477a" Oct 31 23:28:22.388747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.388717 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4"] Oct 31 23:28:22.389226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.389213 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0479fde5-6457-44ad-a1ab-40ad945c563d" containerName="revoke-keys" Oct 31 23:28:22.389226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.389227 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0479fde5-6457-44ad-a1ab-40ad945c563d" containerName="revoke-keys" Oct 31 23:28:22.389329 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.389299 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0479fde5-6457-44ad-a1ab-40ad945c563d" containerName="revoke-keys" Oct 31 23:28:22.392541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.392525 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.398747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.398723 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:28:22.400480 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.400455 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:28:22.402786 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.402765 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4"] Oct 31 23:28:22.533345 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.533315 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp4b\" (UniqueName: \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.533345 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.533348 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.533537 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.533388 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.634063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.634016 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp4b\" (UniqueName: \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.634063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.634059 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.634319 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.634116 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.634693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.634672 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.636426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.636404 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.647071 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.647026 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp4b\" (UniqueName: \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\") pod \"maas-api-revoke-sub-d3fba056-fgwn4\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.707772 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.707746 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:22.984103 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:22.984076 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4"] Oct 31 23:28:22.985704 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:28:22.985669 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f383b5_8aca_4cf5_9dc4_a4ab24b5a76a.slice/crio-f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249 WatchSource:0}: Error finding container f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249: Status 404 returned error can't find the container with id f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249 Oct 31 23:28:23.134390 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:23.134356 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" event={"ID":"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a","Type":"ContainerStarted","Data":"f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249"} Oct 31 23:28:24.142934 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:24.142833 7074 generic.go:356] "Generic (PLEG): container finished" podID="c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" containerID="a3a9a1315f2adafedb5926e54090e4cdefd521e5d53f900a081823d9e1f99f5a" exitCode=0 Oct 31 23:28:24.143429 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:24.143318 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" event={"ID":"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a","Type":"ContainerDied","Data":"a3a9a1315f2adafedb5926e54090e4cdefd521e5d53f900a081823d9e1f99f5a"} Oct 31 23:28:25.723558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.723168 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:25.836033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.812164 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\") pod \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " Oct 31 23:28:25.836033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.812305 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\") pod \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " Oct 31 23:28:25.836033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.812384 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\" (UniqueName: \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\") pod \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\" (UID: \"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a\") " Oct 31 23:28:25.836033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.813239 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca" pod "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" (UID: "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:28:25.836033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.832106 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b" pod "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" (UID: "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a"). InnerVolumeSpecName "kube-api-access-qgp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:28:25.837673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.837335 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca" pod "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" (UID: "c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:28:25.914122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.914076 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:25.914122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.914120 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:25.914375 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:25.914137 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp4b\" (UniqueName: \"kubernetes.io/projected/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a-kube-api-access-qgp4b\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:26.182354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:26.182322 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" event={"ID":"c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a","Type":"ContainerDied","Data":"f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249"} Oct 31 23:28:26.182479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:26.182361 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4" Oct 31 23:28:26.182479 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:26.182374 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07befff62190a8fb01aae87cc15ab9481f1d0714880dd49d587382a10abf249" Oct 31 23:28:44.888459 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:28:44.888423 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:28:45.332577 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.332534 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d"] Oct 31 23:28:45.333107 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.333087 7074 cpu_manager.go:414] "containerMap: removing container" podUID="c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" containerName="revoke-keys" Oct 31 23:28:45.333174 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.333110 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" containerName="revoke-keys" Oct 31 23:28:45.333231 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.333219 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" containerName="revoke-keys" Oct 31 23:28:45.336876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.336849 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.342757 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.342730 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:28:45.345764 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.344243 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:28:45.346234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.346021 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d"] Oct 31 23:28:45.455044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.455001 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.455222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.455120 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzqg\" (UniqueName: \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.455291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.455255 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.556598 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.556560 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzqg\" (UniqueName: \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.556811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.556634 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.556811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.556705 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.557476 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.557455 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.558959 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.558935 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.569015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.568971 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzqg\" (UniqueName: \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\") pod \"maas-api-revoke-sub-ae594979-h4p4d\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.656420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.656334 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:45.931672 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:45.931624 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d"] Oct 31 23:28:45.933275 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:28:45.933251 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa3e304_6cad_4d04_8ead_41de1ebbe393.slice/crio-af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d WatchSource:0}: Error finding container af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d: Status 404 returned error can't find the container with id af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d Oct 31 23:28:46.303202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:46.303167 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" event={"ID":"6aa3e304-6cad-4d04-8ead-41de1ebbe393","Type":"ContainerStarted","Data":"ccd708e8df7889136d45ebaa169a77f8a8706809fb25ce42e47cbfb28d2a1a57"} Oct 31 23:28:46.303318 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:46.303230 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" event={"ID":"6aa3e304-6cad-4d04-8ead-41de1ebbe393","Type":"ContainerStarted","Data":"af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d"} Oct 31 23:28:46.323258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:46.323218 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" podStartSLOduration=1.323206723 podStartE2EDuration="1.323206723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:28:46.323011489 +0000 UTC m=+2304.305753867" watchObservedRunningTime="2026-10-31 23:28:46.323206723 +0000 UTC m=+2304.305949099" Oct 31 23:28:47.309940 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:47.309903 7074 generic.go:356] "Generic (PLEG): container finished" podID="6aa3e304-6cad-4d04-8ead-41de1ebbe393" containerID="ccd708e8df7889136d45ebaa169a77f8a8706809fb25ce42e47cbfb28d2a1a57" exitCode=0 Oct 31 23:28:47.310341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:47.310008 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" event={"ID":"6aa3e304-6cad-4d04-8ead-41de1ebbe393","Type":"ContainerDied","Data":"ccd708e8df7889136d45ebaa169a77f8a8706809fb25ce42e47cbfb28d2a1a57"} Oct 31 23:28:48.453099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.453071 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:48.590335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.590248 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\" (UniqueName: \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\") pod \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " Oct 31 23:28:48.590335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.590303 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\") pod \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " Oct 31 23:28:48.590522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.590376 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\") pod \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\" (UID: \"6aa3e304-6cad-4d04-8ead-41de1ebbe393\") " Oct 31 23:28:48.590969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.590736 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca" pod "6aa3e304-6cad-4d04-8ead-41de1ebbe393" (UID: "6aa3e304-6cad-4d04-8ead-41de1ebbe393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:28:48.592482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.592461 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg" pod "6aa3e304-6cad-4d04-8ead-41de1ebbe393" (UID: "6aa3e304-6cad-4d04-8ead-41de1ebbe393"). InnerVolumeSpecName "kube-api-access-6pzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:28:48.592561 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.592464 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca" pod "6aa3e304-6cad-4d04-8ead-41de1ebbe393" (UID: "6aa3e304-6cad-4d04-8ead-41de1ebbe393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:28:48.692170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.692134 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzqg\" (UniqueName: \"kubernetes.io/projected/6aa3e304-6cad-4d04-8ead-41de1ebbe393-kube-api-access-6pzqg\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:48.692170 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.692168 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:48.692367 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:48.692182 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa3e304-6cad-4d04-8ead-41de1ebbe393-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:28:49.322392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:49.322362 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" event={"ID":"6aa3e304-6cad-4d04-8ead-41de1ebbe393","Type":"ContainerDied","Data":"af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d"} Oct 31 23:28:49.322392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:49.322398 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d" Oct 31 23:28:49.322604 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:28:49.322408 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7ec01baa37ddb7ff167dee9d25a61c4497c9eb67f60dac6c0920e06e276f4d" Oct 31 23:29:03.579112 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.579078 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn"] Oct 31 23:29:03.579601 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.579585 7074 cpu_manager.go:414] "containerMap: removing container" podUID="6aa3e304-6cad-4d04-8ead-41de1ebbe393" containerName="revoke-keys" Oct 31 23:29:03.579682 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.579603 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa3e304-6cad-4d04-8ead-41de1ebbe393" containerName="revoke-keys" Oct 31 23:29:03.579719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.579712 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa3e304-6cad-4d04-8ead-41de1ebbe393" containerName="revoke-keys" Oct 31 23:29:03.583127 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.583106 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.588446 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.588417 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:29:03.588582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.588464 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:29:03.594426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.594396 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn"] Oct 31 23:29:03.659583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.659547 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.659778 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.659590 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.659778 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.659721 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2h5\" (UniqueName: \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.761118 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.761075 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2h5\" (UniqueName: \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.761311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.761234 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.761311 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.761266 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.761903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.761873 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.763673 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.763618 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.773766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.773740 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2h5\" (UniqueName: \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\") pod \"maas-api-revoke-sub-a4aef7d7-9wsqn\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:03.899213 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:03.899137 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:04.221089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:04.221066 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn"] Oct 31 23:29:04.222638 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:29:04.222611 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e33b363_34ee_48c9_950f_04617e8bd8a7.slice/crio-9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031 WatchSource:0}: Error finding container 9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031: Status 404 returned error can't find the container with id 9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031 Oct 31 23:29:04.446356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:04.446321 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" event={"ID":"8e33b363-34ee-48c9-950f-04617e8bd8a7","Type":"ContainerStarted","Data":"9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031"} Oct 31 23:29:05.453245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:05.453150 7074 generic.go:356] "Generic (PLEG): container finished" podID="8e33b363-34ee-48c9-950f-04617e8bd8a7" containerID="8dc30f02ae2b847eea7ef9ebc9bc4cd0806559673b89dc66a69cbff56dca4976" exitCode=0 Oct 31 23:29:05.453245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:05.453236 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" event={"ID":"8e33b363-34ee-48c9-950f-04617e8bd8a7","Type":"ContainerDied","Data":"8dc30f02ae2b847eea7ef9ebc9bc4cd0806559673b89dc66a69cbff56dca4976"} Oct 31 23:29:06.610716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.610688 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:06.694715 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.694678 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\") pod \"8e33b363-34ee-48c9-950f-04617e8bd8a7\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " Oct 31 23:29:06.694905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.694863 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\") pod \"8e33b363-34ee-48c9-950f-04617e8bd8a7\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " Oct 31 23:29:06.694905 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.694895 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\" (UniqueName: \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\") pod \"8e33b363-34ee-48c9-950f-04617e8bd8a7\" (UID: \"8e33b363-34ee-48c9-950f-04617e8bd8a7\") " Oct 31 23:29:06.695274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.695251 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca" pod "8e33b363-34ee-48c9-950f-04617e8bd8a7" (UID: "8e33b363-34ee-48c9-950f-04617e8bd8a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:29:06.696815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.696790 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca" pod "8e33b363-34ee-48c9-950f-04617e8bd8a7" (UID: "8e33b363-34ee-48c9-950f-04617e8bd8a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:29:06.696923 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.696872 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5" pod "8e33b363-34ee-48c9-950f-04617e8bd8a7" (UID: "8e33b363-34ee-48c9-950f-04617e8bd8a7"). InnerVolumeSpecName "kube-api-access-gn2h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:29:06.795889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.795857 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2h5\" (UniqueName: \"kubernetes.io/projected/8e33b363-34ee-48c9-950f-04617e8bd8a7-kube-api-access-gn2h5\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:06.795889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.795885 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:06.795889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:06.795895 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e33b363-34ee-48c9-950f-04617e8bd8a7-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:07.466887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:07.466856 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" Oct 31 23:29:07.467076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:07.466856 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn" event={"ID":"8e33b363-34ee-48c9-950f-04617e8bd8a7","Type":"ContainerDied","Data":"9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031"} Oct 31 23:29:07.467076 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:07.466977 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d567459027c53d27cb4ed1d270bb9ce8a4415bc7653746344c58246b7f4d031" Oct 31 23:29:14.891035 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:29:14.890968 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:29:22.265980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.265939 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c"] Oct 31 23:29:22.266494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.266380 7074 cpu_manager.go:414] "containerMap: removing container" podUID="8e33b363-34ee-48c9-950f-04617e8bd8a7" containerName="revoke-keys" Oct 31 23:29:22.266494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.266392 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e33b363-34ee-48c9-950f-04617e8bd8a7" containerName="revoke-keys" Oct 31 23:29:22.266494 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.266490 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e33b363-34ee-48c9-950f-04617e8bd8a7" containerName="revoke-keys" Oct 31 23:29:22.269724 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.269709 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.277495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.277470 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:29:22.277712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.277560 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:29:22.279556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.279266 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c"] Oct 31 23:29:22.364668 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.364638 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.364876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.364697 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.364876 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.364785 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdq4s\" (UniqueName: \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.466486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.466447 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.466632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.466512 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.466632 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.466538 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdq4s\" (UniqueName: \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.467084 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.467063 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.468812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.468790 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.479072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.479052 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdq4s\" (UniqueName: \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\") pod \"maas-api-revoke-sub-79805ca9-7vt9c\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.588485 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.588404 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:22.863393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:22.863367 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c"] Oct 31 23:29:22.865384 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:29:22.865346 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31993fc_ab75_4b2d_87cd_07b5112d0e9e.slice/crio-2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56 WatchSource:0}: Error finding container 2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56: Status 404 returned error can't find the container with id 2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56 Oct 31 23:29:23.556276 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:23.556238 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" event={"ID":"f31993fc-ab75-4b2d-87cd-07b5112d0e9e","Type":"ContainerStarted","Data":"28a8a1eeaee5eb1179633bb9cdd34915a1aacf51a191aa57a612dd1e5e2faf51"} Oct 31 23:29:23.556668 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:23.556329 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" event={"ID":"f31993fc-ab75-4b2d-87cd-07b5112d0e9e","Type":"ContainerStarted","Data":"2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56"} Oct 31 23:29:23.576812 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:23.576767 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" podStartSLOduration=1.576755199 podStartE2EDuration="1.576755199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:29:23.574331539 +0000 UTC m=+2341.557073917" watchObservedRunningTime="2026-10-31 23:29:23.576755199 +0000 UTC m=+2341.559497605" Oct 31 23:29:24.563739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:24.563700 7074 generic.go:356] "Generic (PLEG): container finished" podID="f31993fc-ab75-4b2d-87cd-07b5112d0e9e" containerID="28a8a1eeaee5eb1179633bb9cdd34915a1aacf51a191aa57a612dd1e5e2faf51" exitCode=0 Oct 31 23:29:24.564164 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:24.563792 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" event={"ID":"f31993fc-ab75-4b2d-87cd-07b5112d0e9e","Type":"ContainerDied","Data":"28a8a1eeaee5eb1179633bb9cdd34915a1aacf51a191aa57a612dd1e5e2faf51"} Oct 31 23:29:25.696758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.696730 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:25.800570 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.800538 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\") pod \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " Oct 31 23:29:25.800744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.800694 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\") pod \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " Oct 31 23:29:25.800744 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.800734 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\" (UniqueName: \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\") pod \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\" (UID: \"f31993fc-ab75-4b2d-87cd-07b5112d0e9e\") " Oct 31 23:29:25.801140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.801111 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca" pod "f31993fc-ab75-4b2d-87cd-07b5112d0e9e" (UID: "f31993fc-ab75-4b2d-87cd-07b5112d0e9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:29:25.802552 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.802526 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca" pod "f31993fc-ab75-4b2d-87cd-07b5112d0e9e" (UID: "f31993fc-ab75-4b2d-87cd-07b5112d0e9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:29:25.802680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.802657 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s" pod "f31993fc-ab75-4b2d-87cd-07b5112d0e9e" (UID: "f31993fc-ab75-4b2d-87cd-07b5112d0e9e"). InnerVolumeSpecName "kube-api-access-zdq4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:29:25.902591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.902503 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:25.902591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.902532 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdq4s\" (UniqueName: \"kubernetes.io/projected/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-kube-api-access-zdq4s\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:25.902591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:25.902545 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31993fc-ab75-4b2d-87cd-07b5112d0e9e-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:26.576832 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:26.576803 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" Oct 31 23:29:26.576832 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:26.576830 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c" event={"ID":"f31993fc-ab75-4b2d-87cd-07b5112d0e9e","Type":"ContainerDied","Data":"2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56"} Oct 31 23:29:26.577132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:26.576893 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf3e2f216c88c664f8a76205a117e17d17528e6b63466727c89968898f98c56" Oct 31 23:29:40.082185 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.082151 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5"] Oct 31 23:29:40.082840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.082643 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f31993fc-ab75-4b2d-87cd-07b5112d0e9e" containerName="revoke-keys" Oct 31 23:29:40.082840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.082656 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31993fc-ab75-4b2d-87cd-07b5112d0e9e" containerName="revoke-keys" Oct 31 23:29:40.082840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.082737 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31993fc-ab75-4b2d-87cd-07b5112d0e9e" containerName="revoke-keys" Oct 31 23:29:40.086281 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.086259 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.092811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.092278 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:29:40.095057 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.093577 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:29:40.098097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.098075 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5"] Oct 31 23:29:40.258191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.258157 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkkt\" (UniqueName: \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.258191 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.258192 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.258423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.258286 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.359293 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.359216 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkkt\" (UniqueName: \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.359293 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.359253 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.359530 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.359304 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.360007 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.359969 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.361676 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.361658 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.371888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.371863 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkkt\" (UniqueName: \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\") pod \"maas-api-revoke-sub-0b7e8edb-2tbc5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.402542 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.402517 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:40.680058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:40.680032 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5"] Oct 31 23:29:40.681737 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:29:40.681707 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc43608d_598e_42e3_86b5_b9c2ca5937a5.slice/crio-0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af WatchSource:0}: Error finding container 0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af: Status 404 returned error can't find the container with id 0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af Oct 31 23:29:41.661163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:41.661131 7074 generic.go:356] "Generic (PLEG): container finished" podID="cc43608d-598e-42e3-86b5-b9c2ca5937a5" containerID="98474b8bfaf8c94e3574c4895d2345d6f81544acb4d47f73679d0b8cf22d7cab" exitCode=0 Oct 31 23:29:41.661533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:41.661190 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" event={"ID":"cc43608d-598e-42e3-86b5-b9c2ca5937a5","Type":"ContainerDied","Data":"98474b8bfaf8c94e3574c4895d2345d6f81544acb4d47f73679d0b8cf22d7cab"} Oct 31 23:29:41.661533 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:41.661225 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" event={"ID":"cc43608d-598e-42e3-86b5-b9c2ca5937a5","Type":"ContainerStarted","Data":"0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af"} Oct 31 23:29:42.800956 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.800933 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:42.884731 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.884696 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\") pod \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " Oct 31 23:29:42.884910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.884753 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\" (UniqueName: \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\") pod \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " Oct 31 23:29:42.884910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.884879 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\") pod \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\" (UID: \"cc43608d-598e-42e3-86b5-b9c2ca5937a5\") " Oct 31 23:29:42.885093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.885071 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca" pod "cc43608d-598e-42e3-86b5-b9c2ca5937a5" (UID: "cc43608d-598e-42e3-86b5-b9c2ca5937a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:29:42.885182 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.885167 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:42.886879 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.886851 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt" pod "cc43608d-598e-42e3-86b5-b9c2ca5937a5" (UID: "cc43608d-598e-42e3-86b5-b9c2ca5937a5"). InnerVolumeSpecName "kube-api-access-dwkkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:29:42.887018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.886904 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca" pod "cc43608d-598e-42e3-86b5-b9c2ca5937a5" (UID: "cc43608d-598e-42e3-86b5-b9c2ca5937a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:29:42.985721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.985645 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkkt\" (UniqueName: \"kubernetes.io/projected/cc43608d-598e-42e3-86b5-b9c2ca5937a5-kube-api-access-dwkkt\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:42.985721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:42.985674 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc43608d-598e-42e3-86b5-b9c2ca5937a5-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:43.673486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:43.673453 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" Oct 31 23:29:43.673486 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:43.673479 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5" event={"ID":"cc43608d-598e-42e3-86b5-b9c2ca5937a5","Type":"ContainerDied","Data":"0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af"} Oct 31 23:29:43.673700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:43.673526 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a463fcacd40c15376374e1dddedbce21e13952a9980b021580e0b8a282dc8af" Oct 31 23:29:44.889622 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:29:44.889584 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:29:50.806912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.806879 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn"] Oct 31 23:29:50.807388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.807374 7074 cpu_manager.go:414] "containerMap: removing container" podUID="cc43608d-598e-42e3-86b5-b9c2ca5937a5" containerName="revoke-keys" Oct 31 23:29:50.807444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.807389 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc43608d-598e-42e3-86b5-b9c2ca5937a5" containerName="revoke-keys" Oct 31 23:29:50.807488 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.807451 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc43608d-598e-42e3-86b5-b9c2ca5937a5" containerName="revoke-keys" Oct 31 23:29:50.810595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.810579 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.818419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.818393 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:29:50.818579 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.818459 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:29:50.821294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.821271 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn"] Oct 31 23:29:50.863887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.863852 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2th\" (UniqueName: \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.863887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.863888 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.864108 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.863979 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.964734 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.964695 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.964902 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.964877 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2th\" (UniqueName: \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.964972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.964909 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.965573 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.965551 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.967134 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.967109 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:50.981203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:50.981174 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2th\" (UniqueName: \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\") pod \"maas-api-revoke-sub-3d4156f1-z7qsn\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:51.125677 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:51.125594 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:51.404409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:51.404380 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn"] Oct 31 23:29:51.405960 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:29:51.405937 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42848faf_2c4c_49cc_8d3e_44dc313841f9.slice/crio-64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106 WatchSource:0}: Error finding container 64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106: Status 404 returned error can't find the container with id 64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106 Oct 31 23:29:51.719426 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:51.719390 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" event={"ID":"42848faf-2c4c-49cc-8d3e-44dc313841f9","Type":"ContainerStarted","Data":"87cbc8df2a58743cd5fdd6f0f44477614edf7fd43bf5b4e7093adc89852adb69"} Oct 31 23:29:51.719582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:51.719448 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" event={"ID":"42848faf-2c4c-49cc-8d3e-44dc313841f9","Type":"ContainerStarted","Data":"64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106"} Oct 31 23:29:51.739815 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:51.739768 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" podStartSLOduration=1.739753911 podStartE2EDuration="1.739753911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:29:51.739050915 +0000 UTC m=+2369.721793293" watchObservedRunningTime="2026-10-31 23:29:51.739753911 +0000 UTC m=+2369.722496288" Oct 31 23:29:52.726497 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:52.726462 7074 generic.go:356] "Generic (PLEG): container finished" podID="42848faf-2c4c-49cc-8d3e-44dc313841f9" containerID="87cbc8df2a58743cd5fdd6f0f44477614edf7fd43bf5b4e7093adc89852adb69" exitCode=0 Oct 31 23:29:52.727063 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:52.726544 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" event={"ID":"42848faf-2c4c-49cc-8d3e-44dc313841f9","Type":"ContainerDied","Data":"87cbc8df2a58743cd5fdd6f0f44477614edf7fd43bf5b4e7093adc89852adb69"} Oct 31 23:29:53.878052 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.878030 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:53.996544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.996444 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\") pod \"42848faf-2c4c-49cc-8d3e-44dc313841f9\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " Oct 31 23:29:53.996544 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.996509 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\") pod \"42848faf-2c4c-49cc-8d3e-44dc313841f9\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " Oct 31 23:29:53.996790 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.996672 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\" (UniqueName: \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\") pod \"42848faf-2c4c-49cc-8d3e-44dc313841f9\" (UID: \"42848faf-2c4c-49cc-8d3e-44dc313841f9\") " Oct 31 23:29:53.996945 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.996913 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca" pod "42848faf-2c4c-49cc-8d3e-44dc313841f9" (UID: "42848faf-2c4c-49cc-8d3e-44dc313841f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:29:53.997120 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.997103 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:53.998737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.998714 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th" pod "42848faf-2c4c-49cc-8d3e-44dc313841f9" (UID: "42848faf-2c4c-49cc-8d3e-44dc313841f9"). InnerVolumeSpecName "kube-api-access-kv2th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:29:53.998808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:53.998779 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca" pod "42848faf-2c4c-49cc-8d3e-44dc313841f9" (UID: "42848faf-2c4c-49cc-8d3e-44dc313841f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:29:54.098101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:54.098069 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2th\" (UniqueName: \"kubernetes.io/projected/42848faf-2c4c-49cc-8d3e-44dc313841f9-kube-api-access-kv2th\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:54.098101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:54.098097 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42848faf-2c4c-49cc-8d3e-44dc313841f9-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:29:54.739046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:54.739014 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" Oct 31 23:29:54.739199 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:54.739042 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn" event={"ID":"42848faf-2c4c-49cc-8d3e-44dc313841f9","Type":"ContainerDied","Data":"64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106"} Oct 31 23:29:54.739199 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:29:54.739087 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a842c7f59fc78529bc46d681b6f3969c633feaaefef92635459828ab3b6106" Oct 31 23:30:00.143297 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.143263 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:00.143844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.143748 7074 cpu_manager.go:414] "containerMap: removing container" podUID="42848faf-2c4c-49cc-8d3e-44dc313841f9" containerName="revoke-keys" Oct 31 23:30:00.143844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.143760 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42848faf-2c4c-49cc-8d3e-44dc313841f9" containerName="revoke-keys" Oct 31 23:30:00.143844 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.143836 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="42848faf-2c4c-49cc-8d3e-44dc313841f9" containerName="revoke-keys" Oct 31 23:30:00.147341 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.147318 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.155614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.154223 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:30:00.156525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.156497 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:00.263256 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.263203 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr78c\" (UniqueName: \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.263562 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.263535 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.365017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.364972 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.365189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.365032 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr78c\" (UniqueName: \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.365582 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.365564 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.377831 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.377802 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr78c\" (UniqueName: \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\") pod \"maas-api-key-cleanup-29891490-zt4fn\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.468465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.468389 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:00.743438 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.743411 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:00.744902 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:30:00.744869 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88520f7f_d842_4064_97d7_9b4d67f4b5d3.slice/crio-d864037444e2ac1f04457dc19a186120eb0c93a9aa8d3572281d92b80860449d WatchSource:0}: Error finding container d864037444e2ac1f04457dc19a186120eb0c93a9aa8d3572281d92b80860449d: Status 404 returned error can't find the container with id d864037444e2ac1f04457dc19a186120eb0c93a9aa8d3572281d92b80860449d Oct 31 23:30:00.774854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:00.774825 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerStarted","Data":"d864037444e2ac1f04457dc19a186120eb0c93a9aa8d3572281d92b80860449d"} Oct 31 23:30:01.782767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:01.782733 7074 generic.go:356] "Generic (PLEG): container finished" podID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerID="9896decfe6a3e6696b72d9a9dd4371d83e0e932a81d9af013f6c034e136e98ac" exitCode=60 Oct 31 23:30:01.783162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:01.782817 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerDied","Data":"9896decfe6a3e6696b72d9a9dd4371d83e0e932a81d9af013f6c034e136e98ac"} Oct 31 23:30:01.783162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:01.783059 7074 scope.go:122] "RemoveContainer" containerID="9896decfe6a3e6696b72d9a9dd4371d83e0e932a81d9af013f6c034e136e98ac" Oct 31 23:30:02.791267 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:02.791234 7074 generic.go:356] "Generic (PLEG): container finished" podID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerID="426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e" exitCode=60 Oct 31 23:30:02.791716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:02.791324 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerDied","Data":"426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e"} Oct 31 23:30:02.791716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:02.791392 7074 scope.go:122] "RemoveContainer" containerID="9896decfe6a3e6696b72d9a9dd4371d83e0e932a81d9af013f6c034e136e98ac" Oct 31 23:30:02.791716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:02.791646 7074 scope.go:122] "RemoveContainer" containerID="426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e" Oct 31 23:30:02.791962 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:30:02.791937 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891490-zt4fn_odh-ai-gateway-infra(88520f7f-d842-4064-97d7-9b4d67f4b5d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" Oct 31 23:30:03.818001 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:03.817950 7074 scope.go:122] "RemoveContainer" containerID="426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e" Oct 31 23:30:03.818493 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:30:03.818293 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891490-zt4fn_odh-ai-gateway-infra(88520f7f-d842-4064-97d7-9b4d67f4b5d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" Oct 31 23:30:12.161691 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:12.161646 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn"] Oct 31 23:30:12.167100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:12.167066 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7qsn"] Oct 31 23:30:14.141376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:14.141335 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42848faf-2c4c-49cc-8d3e-44dc313841f9" path="/var/lib/kubelet/pods/42848faf-2c4c-49cc-8d3e-44dc313841f9/volumes" Oct 31 23:30:14.887207 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:30:14.887174 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:30:17.134700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:17.134662 7074 scope.go:122] "RemoveContainer" containerID="426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e" Oct 31 23:30:17.895087 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:17.895049 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerStarted","Data":"ab8ab3bd8bae0b0d333f4e0056a4b07f4c8eb4a7917f6518906cbf7cef7cec48"} Oct 31 23:30:17.915469 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:17.915418 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" podStartSLOduration=17.915404568 podStartE2EDuration="17.915404568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:30:17.913486294 +0000 UTC m=+2395.896228671" watchObservedRunningTime="2026-10-31 23:30:17.915404568 +0000 UTC m=+2395.898146965" Oct 31 23:30:18.902515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:18.902484 7074 generic.go:356] "Generic (PLEG): container finished" podID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerID="ab8ab3bd8bae0b0d333f4e0056a4b07f4c8eb4a7917f6518906cbf7cef7cec48" exitCode=60 Oct 31 23:30:18.902930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:18.902562 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerDied","Data":"ab8ab3bd8bae0b0d333f4e0056a4b07f4c8eb4a7917f6518906cbf7cef7cec48"} Oct 31 23:30:18.902930 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:18.902632 7074 scope.go:122] "RemoveContainer" containerID="426ac3dc074f27652584d036382b0d6a5cb0a0cfa1d76f09ff6cca831f74de9e" Oct 31 23:30:18.903058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:18.903022 7074 scope.go:122] "RemoveContainer" containerID="ab8ab3bd8bae0b0d333f4e0056a4b07f4c8eb4a7917f6518906cbf7cef7cec48" Oct 31 23:30:18.903332 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:30:18.903310 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891490-zt4fn_odh-ai-gateway-infra(88520f7f-d842-4064-97d7-9b4d67f4b5d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" Oct 31 23:30:18.924505 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:18.924481 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:20.051536 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.051508 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:20.135803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.135775 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\" (UniqueName: \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\") pod \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " Oct 31 23:30:20.135982 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.135891 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\") pod \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\" (UID: \"88520f7f-d842-4064-97d7-9b4d67f4b5d3\") " Oct 31 23:30:20.136277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.136255 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca" pod "88520f7f-d842-4064-97d7-9b4d67f4b5d3" (UID: "88520f7f-d842-4064-97d7-9b4d67f4b5d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:30:20.138102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.138078 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c" pod "88520f7f-d842-4064-97d7-9b4d67f4b5d3" (UID: "88520f7f-d842-4064-97d7-9b4d67f4b5d3"). InnerVolumeSpecName "kube-api-access-wr78c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:20.238127 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.238038 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88520f7f-d842-4064-97d7-9b4d67f4b5d3-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:20.238127 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.238067 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr78c\" (UniqueName: \"kubernetes.io/projected/88520f7f-d842-4064-97d7-9b4d67f4b5d3-kube-api-access-wr78c\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:20.917405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.917379 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" Oct 31 23:30:20.917405 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.917393 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn" event={"ID":"88520f7f-d842-4064-97d7-9b4d67f4b5d3","Type":"ContainerDied","Data":"d864037444e2ac1f04457dc19a186120eb0c93a9aa8d3572281d92b80860449d"} Oct 31 23:30:20.917635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.917448 7074 scope.go:122] "RemoveContainer" containerID="ab8ab3bd8bae0b0d333f4e0056a4b07f4c8eb4a7917f6518906cbf7cef7cec48" Oct 31 23:30:20.936028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.935985 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:20.941350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:20.941324 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891490-zt4fn"] Oct 31 23:30:22.141978 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:22.141943 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" path="/var/lib/kubelet/pods/88520f7f-d842-4064-97d7-9b4d67f4b5d3/volumes" Oct 31 23:30:22.598062 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:22.598034 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:30:22.598259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:30:22.598170 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:30:44.888545 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:30:44.888503 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:31:14.888788 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:31:14.888704 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:31:26.024614 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:31:26.024582 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp"] Oct 31 23:31:26.027708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:31:26.027679 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6mxdp"] Oct 31 23:31:26.141733 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:31:26.141696 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223a4404-15dd-4a6d-970d-31f3f05f4ff0" path="/var/lib/kubelet/pods/223a4404-15dd-4a6d-970d-31f3f05f4ff0/volumes" Oct 31 23:31:44.887716 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:31:44.887677 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:32:14.887453 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:32:14.887419 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:32:19.024209 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:19.024162 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh"] Oct 31 23:32:19.026502 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:19.026474 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-prmvh"] Oct 31 23:32:20.141816 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:20.141785 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d83fb75-70a2-4329-b759-4ece9c4ce55d" path="/var/lib/kubelet/pods/1d83fb75-70a2-4329-b759-4ece9c4ce55d/volumes" Oct 31 23:32:23.169104 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:23.169068 7074 scope.go:122] "RemoveContainer" containerID="15ec22812464cd8a6b0e5cf1541cd451f92b42318941cdea59b2a656f9f0fa9f" Oct 31 23:32:38.023041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:38.023002 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm"] Oct 31 23:32:38.025531 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:38.025504 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-59smm"] Oct 31 23:32:38.141961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:38.141925 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df9042d-2469-4337-9c47-ebf03774d693" path="/var/lib/kubelet/pods/1df9042d-2469-4337-9c47-ebf03774d693/volumes" Oct 31 23:32:44.888205 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:32:44.888167 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:32:49.023222 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:49.023185 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88"] Oct 31 23:32:49.026291 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:49.026265 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-scx88"] Oct 31 23:32:50.141089 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:32:50.141054 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255046e8-664c-4cb2-b2da-c3b70e7126eb" path="/var/lib/kubelet/pods/255046e8-664c-4cb2-b2da-c3b70e7126eb/volumes" Oct 31 23:33:14.888368 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:33:14.888329 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:33:16.022245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:16.022200 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7"] Oct 31 23:33:16.024372 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:16.024348 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7s9j7"] Oct 31 23:33:16.150252 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:16.150219 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0479fde5-6457-44ad-a1ab-40ad945c563d" path="/var/lib/kubelet/pods/0479fde5-6457-44ad-a1ab-40ad945c563d/volumes" Oct 31 23:33:23.191438 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:23.191401 7074 scope.go:122] "RemoveContainer" containerID="1b3358e5fe43e7c27b6267f12c4f865c385c40b325e43a8bc2ce0bff751195cf" Oct 31 23:33:26.022504 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:26.022465 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4"] Oct 31 23:33:26.025195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:26.025173 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-fgwn4"] Oct 31 23:33:26.140924 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:26.140889 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a" path="/var/lib/kubelet/pods/c3f383b5-8aca-4cf5-9dc4-a4ab24b5a76a/volumes" Oct 31 23:33:44.888578 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:33:44.888537 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:33:49.022036 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:49.021984 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d"] Oct 31 23:33:49.026740 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:49.026714 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h4p4d"] Oct 31 23:33:50.141409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:33:50.141364 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa3e304-6cad-4d04-8ead-41de1ebbe393" path="/var/lib/kubelet/pods/6aa3e304-6cad-4d04-8ead-41de1ebbe393/volumes" Oct 31 23:34:07.022254 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:07.022154 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn"] Oct 31 23:34:07.024580 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:07.024560 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9wsqn"] Oct 31 23:34:08.141072 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:08.141027 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e33b363-34ee-48c9-950f-04617e8bd8a7" path="/var/lib/kubelet/pods/8e33b363-34ee-48c9-950f-04617e8bd8a7/volumes" Oct 31 23:34:14.888182 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:34:14.888145 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:34:23.215658 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:23.215615 7074 scope.go:122] "RemoveContainer" containerID="bad5e87f4205b832cd1b93531f44c8bacd65725ccc2ff738b6cca9821426b8f8" Oct 31 23:34:23.224936 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:23.224909 7074 scope.go:122] "RemoveContainer" containerID="0bd0351a78db73a94275610cc64f2d3afda7a9a2bde90ae9f465be6ce2cf9c69" Oct 31 23:34:23.233510 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:23.233488 7074 scope.go:122] "RemoveContainer" containerID="a3a9a1315f2adafedb5926e54090e4cdefd521e5d53f900a081823d9e1f99f5a" Oct 31 23:34:23.242396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:23.242370 7074 scope.go:122] "RemoveContainer" containerID="ea8beee6dae8c89bcc0b9b1846ed4a198dc0eddd665fd8d84381bde6503824bd" Oct 31 23:34:26.021046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:26.021008 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c"] Oct 31 23:34:26.023839 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:26.023814 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7vt9c"] Oct 31 23:34:26.141420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:26.141388 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31993fc-ab75-4b2d-87cd-07b5112d0e9e" path="/var/lib/kubelet/pods/f31993fc-ab75-4b2d-87cd-07b5112d0e9e/volumes" Oct 31 23:34:43.027939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:43.027901 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5"] Oct 31 23:34:43.030821 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:43.030790 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2tbc5"] Oct 31 23:34:44.141806 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:34:44.141759 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc43608d-598e-42e3-86b5-b9c2ca5937a5" path="/var/lib/kubelet/pods/cc43608d-598e-42e3-86b5-b9c2ca5937a5/volumes" Oct 31 23:34:44.888898 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:34:44.888860 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:35:14.888463 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:35:14.888423 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:35:22.642695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:35:22.642662 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:35:22.644244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:35:22.644221 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:35:23.276778 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:35:23.276741 7074 scope.go:122] "RemoveContainer" containerID="28a8a1eeaee5eb1179633bb9cdd34915a1aacf51a191aa57a612dd1e5e2faf51" Oct 31 23:35:23.286158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:35:23.286139 7074 scope.go:122] "RemoveContainer" containerID="ccd708e8df7889136d45ebaa169a77f8a8706809fb25ce42e47cbfb28d2a1a57" Oct 31 23:35:23.294347 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:35:23.294328 7074 scope.go:122] "RemoveContainer" containerID="8dc30f02ae2b847eea7ef9ebc9bc4cd0806559673b89dc66a69cbff56dca4976" Oct 31 23:35:44.888965 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:35:44.888926 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:36:14.888166 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:36:14.888123 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:36:23.321862 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:23.321821 7074 scope.go:122] "RemoveContainer" containerID="87cbc8df2a58743cd5fdd6f0f44477614edf7fd43bf5b4e7093adc89852adb69" Oct 31 23:36:23.330870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:23.330845 7074 scope.go:122] "RemoveContainer" containerID="98474b8bfaf8c94e3574c4895d2345d6f81544acb4d47f73679d0b8cf22d7cab" Oct 31 23:36:34.798030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.797919 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:36:34.798591 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.798288 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" containerID="cri-o://17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3" gracePeriod=30 Oct 31 23:36:34.800680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.800289 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:36:34.800680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.800515 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" containerID="cri-o://63fa1df26e9728d0a5ddca850afc17c30cab3803c1bc6e84c985f9f3c6021880" gracePeriod=30 Oct 31 23:36:34.803391 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.803366 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:36:34.803620 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:34.803561 7074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" containerID="cri-o://3654dca51588c088a8d1c1810d2c52532d8c536a0ca68e79c39fc99aefcc2000" gracePeriod=30 Oct 31 23:36:40.068650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.068617 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:36:40.099045 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.098933 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099059 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099108 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099145 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099179 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099244 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099235 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099268 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099304 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:40.099526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099319 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099526 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099422 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\") pod \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\" (UID: \"7270a3ff-1d40-4ab1-b7f6-f6178e060405\") " Oct 31 23:36:40.099753 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099544 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.099810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099776 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.099926 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099909 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.100037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099930 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.100037 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.099945 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.101716 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.101685 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.103323 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.103105 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.104156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.104138 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.106470 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.106449 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "kube-api-access-b5hhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.106889 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.106864 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.106974 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.106957 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo" pod "7270a3ff-1d40-4ab1-b7f6-f6178e060405" (UID: "7270a3ff-1d40-4ab1-b7f6-f6178e060405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:40.119425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.118623 7074 generic.go:356] "Generic (PLEG): container finished" podID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerID="17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3" exitCode=0 Oct 31 23:36:40.119425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.118649 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" Oct 31 23:36:40.119425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.118762 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" event={"ID":"7270a3ff-1d40-4ab1-b7f6-f6178e060405","Type":"ContainerDied","Data":"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3"} Oct 31 23:36:40.119425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.118843 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd" event={"ID":"7270a3ff-1d40-4ab1-b7f6-f6178e060405","Type":"ContainerDied","Data":"733ced5ffa8c90785b95b57768c361b34948932a3afebf2a4596f2d6c0f210f3"} Oct 31 23:36:40.119425 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.118879 7074 scope.go:122] "RemoveContainer" containerID="17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3" Oct 31 23:36:40.129679 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.129656 7074 generic.go:356] "Generic (PLEG): container finished" podID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerID="3654dca51588c088a8d1c1810d2c52532d8c536a0ca68e79c39fc99aefcc2000" exitCode=0 Oct 31 23:36:40.129818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.129741 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" event={"ID":"2c529ca5-f36e-4c77-a315-e5ffa4609612","Type":"ContainerDied","Data":"3654dca51588c088a8d1c1810d2c52532d8c536a0ca68e79c39fc99aefcc2000"} Oct 31 23:36:40.131455 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.131425 7074 generic.go:356] "Generic (PLEG): container finished" podID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerID="63fa1df26e9728d0a5ddca850afc17c30cab3803c1bc6e84c985f9f3c6021880" exitCode=0 Oct 31 23:36:40.131670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.131617 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" event={"ID":"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74","Type":"ContainerDied","Data":"63fa1df26e9728d0a5ddca850afc17c30cab3803c1bc6e84c985f9f3c6021880"} Oct 31 23:36:40.143381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.143345 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:36:40.145849 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.131716 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" event={"ID":"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74","Type":"ContainerDied","Data":"c8cd0dc2604a922ddb2cbbf28406997c96b4518ae7db8955caf40f9c2663309c"} Oct 31 23:36:40.146379 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.146276 7074 scope.go:122] "RemoveContainer" containerID="17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3" Oct 31 23:36:40.149457 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:36:40.147491 7074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3\": container with ID starting with 17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3 not found: ID does not exist" containerID="17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3" Oct 31 23:36:40.149457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.147528 7074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3"} err="failed to get container status \"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3\": rpc error: code = NotFound desc = could not find container \"17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3\": container with ID starting with 17d93c7e09b3051d479a7b5d34526b57afea36fc0c1cb45710cd570ad7783bd3 not found: ID does not exist" Oct 31 23:36:40.149457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.147553 7074 scope.go:122] "RemoveContainer" containerID="63fa1df26e9728d0a5ddca850afc17c30cab3803c1bc6e84c985f9f3c6021880" Oct 31 23:36:40.155135 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.155114 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:36:40.187307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.187279 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:36:40.194389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.194362 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dvbhd"] Oct 31 23:36:40.201013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.200976 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201039 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201086 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201125 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201160 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201197 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201240 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201278 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201316 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201578 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201350 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201578 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201466 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201578 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201541 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201575 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201609 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201639 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201735 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201692 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.201981 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201732 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\") pod \"2c529ca5-f36e-4c77-a315-e5ffa4609612\" (UID: \"2c529ca5-f36e-4c77-a315-e5ffa4609612\") " Oct 31 23:36:40.201981 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201791 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\") pod \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\" (UID: \"94a4b5f6-7b12-4e65-a63c-88d7c06c2c74\") " Oct 31 23:36:40.202266 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202242 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202275 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hhj\" (UniqueName: \"kubernetes.io/projected/7270a3ff-1d40-4ab1-b7f6-f6178e060405-kube-api-access-b5hhj\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202291 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202307 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202321 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7270a3ff-1d40-4ab1-b7f6-f6178e060405-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202334 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7270a3ff-1d40-4ab1-b7f6-f6178e060405-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.202612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201394 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201661 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.201856 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202061 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:40.202612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202374 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202671 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202805 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202877 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.202820 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.203102 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.203078 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.203218 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.203193 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:40.204309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.204281 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "kube-api-access-96shw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.205313 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.205286 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "kube-api-access-8hn5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.205396 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.205367 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:40.205847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.205821 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.206309 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.206289 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy" pod "2c529ca5-f36e-4c77-a315-e5ffa4609612" (UID: "2c529ca5-f36e-4c77-a315-e5ffa4609612"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.206401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.206325 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:40.206401 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.206379 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.208674 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.208650 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy" pod "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" (UID: "94a4b5f6-7b12-4e65-a63c-88d7c06c2c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.303597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303556 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303587 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hn5l\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-kube-api-access-8hn5l\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303597 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303606 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303618 7074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-token\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303628 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303637 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303646 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303655 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303665 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303673 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c529ca5-f36e-4c77-a315-e5ffa4609612-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303681 7074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-envoy\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303689 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96shw\" (UniqueName: \"kubernetes.io/projected/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-kube-api-access-96shw\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303698 7074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-data\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303706 7074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c529ca5-f36e-4c77-a315-e5ffa4609612-istiod-ca-cert\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303713 7074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-credential-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303721 7074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-istio-podinfo\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303729 7074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c529ca5-f36e-4c77-a315-e5ffa4609612-workload-certs\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.303843 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:40.303737 7074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74-workload-socket\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:41.140038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.139939 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" Oct 31 23:36:41.140038 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.139933 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp" event={"ID":"2c529ca5-f36e-4c77-a315-e5ffa4609612","Type":"ContainerDied","Data":"bdcbd3667ddcdfbda42c239fcc475b964164a5d4e13dec883ac95c0920359e2a"} Oct 31 23:36:41.140595 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.140059 7074 scope.go:122] "RemoveContainer" containerID="3654dca51588c088a8d1c1810d2c52532d8c536a0ca68e79c39fc99aefcc2000" Oct 31 23:36:41.140816 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.140798 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd" Oct 31 23:36:41.168592 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.168565 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:36:41.174879 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.174855 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p76fd"] Oct 31 23:36:41.186581 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.186556 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:36:41.191106 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:41.191082 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mcdvp"] Oct 31 23:36:42.142158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:42.142129 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" path="/var/lib/kubelet/pods/2c529ca5-f36e-4c77-a315-e5ffa4609612/volumes" Oct 31 23:36:42.142600 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:42.142586 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" path="/var/lib/kubelet/pods/7270a3ff-1d40-4ab1-b7f6-f6178e060405/volumes" Oct 31 23:36:42.143060 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:42.143040 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" path="/var/lib/kubelet/pods/94a4b5f6-7b12-4e65-a63c-88d7c06c2c74/volumes" Oct 31 23:36:44.888160 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:36:44.888118 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:36:49.820131 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820089 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g"] Oct 31 23:36:49.820896 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820869 7074 cpu_manager.go:414] "containerMap: removing container" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.820896 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820895 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820928 7074 cpu_manager.go:414] "containerMap: removing container" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820936 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820958 7074 cpu_manager.go:414] "containerMap: removing container" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.820969 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821026 7074 cpu_manager.go:414] "containerMap: removing container" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821034 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821054 7074 cpu_manager.go:414] "containerMap: removing container" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821061 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821072 7074 cpu_manager.go:414] "containerMap: removing container" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" Oct 31 23:36:49.821128 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821079 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" Oct 31 23:36:49.821523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821195 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821210 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a4b5f6-7b12-4e65-a63c-88d7c06c2c74" containerName="istio-proxy" Oct 31 23:36:49.821523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821220 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:36:49.821523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821227 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c529ca5-f36e-4c77-a315-e5ffa4609612" containerName="istio-proxy" Oct 31 23:36:49.821523 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.821233 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7270a3ff-1d40-4ab1-b7f6-f6178e060405" containerName="istio-proxy" Oct 31 23:36:49.826611 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.826591 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.837820 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.837789 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g"] Oct 31 23:36:49.904159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904117 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904165 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchh4\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-kube-api-access-hchh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904244 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904281 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904352 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904372 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904399 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904545 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.904670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.904567 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb1470c-1020-463d-a1ef-d00399fb13d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:49.917002 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:49.916959 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nf6c6\"" Oct 31 23:36:50.005296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005259 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005301 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005337 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005419 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005447 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb1470c-1020-463d-a1ef-d00399fb13d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005512 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005559 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005535 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchh4\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-kube-api-access-hchh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005566 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005808 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005588 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005910 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005888 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.005972 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.005944 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.006258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.006232 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.006353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.006330 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb1470c-1020-463d-a1ef-d00399fb13d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.006466 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.006446 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.007880 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.007819 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.008473 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.008444 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.022294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.022256 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.022457 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.022433 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchh4\" (UniqueName: \"kubernetes.io/projected/3bb1470c-1020-463d-a1ef-d00399fb13d4-kube-api-access-hchh4\") pod \"maas-default-gateway-openshift-default-6886b96f9-m522g\" (UID: \"3bb1470c-1020-463d-a1ef-d00399fb13d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.146708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.146613 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:50.433633 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.433604 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g"] Oct 31 23:36:50.435702 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:36:50.435661 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb1470c_1020_463d_a1ef_d00399fb13d4.slice/crio-dba5ecc48dca26891cc62104daf0f9ad09c8f697e4da9191a7dc3b056317f49f WatchSource:0}: Error finding container dba5ecc48dca26891cc62104daf0f9ad09c8f697e4da9191a7dc3b056317f49f: Status 404 returned error can't find the container with id dba5ecc48dca26891cc62104daf0f9ad09c8f697e4da9191a7dc3b056317f49f Oct 31 23:36:50.437954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.437920 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:36:50.438051 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.438020 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:36:50.438100 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:50.438052 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:36:51.199150 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:51.199114 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" event={"ID":"3bb1470c-1020-463d-a1ef-d00399fb13d4","Type":"ContainerStarted","Data":"2be2a5ff38a2293ee9fc8f87a20e75c8baf768160e0feacce3a2dea34c29c4ea"} Oct 31 23:36:51.199541 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:51.199174 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" event={"ID":"3bb1470c-1020-463d-a1ef-d00399fb13d4","Type":"ContainerStarted","Data":"dba5ecc48dca26891cc62104daf0f9ad09c8f697e4da9191a7dc3b056317f49f"} Oct 31 23:36:51.226107 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:51.226061 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podStartSLOduration=2.226047628 podStartE2EDuration="2.226047628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:51.224580998 +0000 UTC m=+2789.207323375" watchObservedRunningTime="2026-10-31 23:36:51.226047628 +0000 UTC m=+2789.208790004" Oct 31 23:36:52.157018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:52.154911 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:52.157018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:52.155067 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:52.157018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:52.155111 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:53.151016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:53.147418 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:53.151016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:53.147480 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:54.165014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:54.163449 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:54.165014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:54.163517 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:55.147393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:55.147356 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:55.147713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:55.147681 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:56.160543 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:56.160503 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:56.161175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:56.161135 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:57.147147 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:57.147111 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 23:36:57.147331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:57.147173 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 23:36:58.543306 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:58.543273 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:58.543807 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:58.543646 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:36:58.545538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:36:58.545514 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" Oct 31 23:37:14.886362 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:37:14.886280 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:37:19.244325 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.244253 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8"] Oct 31 23:37:19.245092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.245062 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="88520f7f-d842-4064-97d7-9b4d67f4b5d3" containerName="cleanup" Oct 31 23:37:19.248589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.248573 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.254941 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.254915 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:37:19.256637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.256496 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:37:19.262243 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.262218 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8"] Oct 31 23:37:19.352196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.352155 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfhd\" (UniqueName: \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.352196 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.352199 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.352393 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.352323 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.453700 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.453664 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfhd\" (UniqueName: \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.453918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.453711 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.453918 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.453754 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.454449 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.454424 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.456359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.456325 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.466353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.466322 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfhd\" (UniqueName: \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\") pod \"maas-api-revoke-sub-7b2ce870-r5kk8\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.566726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.566688 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:19.853143 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:19.853116 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8"] Oct 31 23:37:19.855384 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:37:19.855357 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe30110f_43c6_4a67_ad18_1ebb01287ba5.slice/crio-507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198 WatchSource:0}: Error finding container 507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198: Status 404 returned error can't find the container with id 507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198 Oct 31 23:37:20.360933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:20.360898 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" event={"ID":"fe30110f-43c6-4a67-ad18-1ebb01287ba5","Type":"ContainerStarted","Data":"aaf507a4552898f209e6a9614f3d6f677790ea25d158fc617b39de73adbc76e8"} Oct 31 23:37:20.361327 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:20.360975 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" event={"ID":"fe30110f-43c6-4a67-ad18-1ebb01287ba5","Type":"ContainerStarted","Data":"507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198"} Oct 31 23:37:20.380074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:20.380025 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" podStartSLOduration=1.379984686 podStartE2EDuration="1.379984686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:37:20.379377197 +0000 UTC m=+2818.362119575" watchObservedRunningTime="2026-10-31 23:37:20.379984686 +0000 UTC m=+2818.362727059" Oct 31 23:37:21.369441 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:21.369406 7074 generic.go:356] "Generic (PLEG): container finished" podID="fe30110f-43c6-4a67-ad18-1ebb01287ba5" containerID="aaf507a4552898f209e6a9614f3d6f677790ea25d158fc617b39de73adbc76e8" exitCode=0 Oct 31 23:37:21.369897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:21.369489 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" event={"ID":"fe30110f-43c6-4a67-ad18-1ebb01287ba5","Type":"ContainerDied","Data":"aaf507a4552898f209e6a9614f3d6f677790ea25d158fc617b39de73adbc76e8"} Oct 31 23:37:22.509376 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.509349 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:22.585348 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.585318 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\" (UniqueName: \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\") pod \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " Oct 31 23:37:22.585539 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.585473 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\") pod \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " Oct 31 23:37:22.585596 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.585534 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\") pod \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\" (UID: \"fe30110f-43c6-4a67-ad18-1ebb01287ba5\") " Oct 31 23:37:22.585840 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.585804 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca" pod "fe30110f-43c6-4a67-ad18-1ebb01287ba5" (UID: "fe30110f-43c6-4a67-ad18-1ebb01287ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:37:22.587540 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.587519 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca" pod "fe30110f-43c6-4a67-ad18-1ebb01287ba5" (UID: "fe30110f-43c6-4a67-ad18-1ebb01287ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:37:22.587650 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.587539 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd" pod "fe30110f-43c6-4a67-ad18-1ebb01287ba5" (UID: "fe30110f-43c6-4a67-ad18-1ebb01287ba5"). InnerVolumeSpecName "kube-api-access-jlfhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:22.686763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.686665 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfhd\" (UniqueName: \"kubernetes.io/projected/fe30110f-43c6-4a67-ad18-1ebb01287ba5-kube-api-access-jlfhd\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:22.686763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.686703 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:22.686763 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:22.686715 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe30110f-43c6-4a67-ad18-1ebb01287ba5-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:23.380851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:23.380786 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" event={"ID":"fe30110f-43c6-4a67-ad18-1ebb01287ba5","Type":"ContainerDied","Data":"507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198"} Oct 31 23:37:23.380851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:23.380849 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507c753ab76edce3bb532f9073ecb1da8febd94d6e933693b97d901165a48198" Oct 31 23:37:23.381098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:23.380807 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8" Oct 31 23:37:30.282586 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.282550 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f"] Oct 31 23:37:30.283192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.283174 7074 cpu_manager.go:414] "containerMap: removing container" podUID="fe30110f-43c6-4a67-ad18-1ebb01287ba5" containerName="revoke-keys" Oct 31 23:37:30.283242 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.283194 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe30110f-43c6-4a67-ad18-1ebb01287ba5" containerName="revoke-keys" Oct 31 23:37:30.283303 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.283286 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe30110f-43c6-4a67-ad18-1ebb01287ba5" containerName="revoke-keys" Oct 31 23:37:30.287921 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.287902 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.294499 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.294464 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:37:30.294613 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.294516 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:37:30.300427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.297246 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f"] Oct 31 23:37:30.469296 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.469259 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.469482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.469342 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.469482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.469444 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wcx\" (UniqueName: \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.570803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.570709 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.570803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.570797 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wcx\" (UniqueName: \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.571013 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.570844 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.571517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.571492 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.573101 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.573084 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.583228 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.583204 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wcx\" (UniqueName: \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.603781 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.603755 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:30.885792 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:30.885763 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f"] Oct 31 23:37:30.887576 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:37:30.887550 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e751995_61e6_44be_a827_81f1e8b25b39.slice/crio-75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83 WatchSource:0}: Error finding container 75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83: Status 404 returned error can't find the container with id 75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83 Oct 31 23:37:31.426714 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:31.426627 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" event={"ID":"8e751995-61e6-44be-a827-81f1e8b25b39","Type":"ContainerStarted","Data":"2a9517a54804e0ca505ba1483f713d8e332df2b9074cc821534bf6803be1d227"} Oct 31 23:37:31.426714 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:31.426678 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" event={"ID":"8e751995-61e6-44be-a827-81f1e8b25b39","Type":"ContainerStarted","Data":"75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83"} Oct 31 23:37:31.448717 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:31.448663 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" podStartSLOduration=1.448650066 podStartE2EDuration="1.448650066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:37:31.447546222 +0000 UTC m=+2829.430288624" watchObservedRunningTime="2026-10-31 23:37:31.448650066 +0000 UTC m=+2829.431392441" Oct 31 23:37:32.433517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:32.433480 7074 generic.go:356] "Generic (PLEG): container finished" podID="8e751995-61e6-44be-a827-81f1e8b25b39" containerID="2a9517a54804e0ca505ba1483f713d8e332df2b9074cc821534bf6803be1d227" exitCode=0 Oct 31 23:37:32.434014 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:32.433567 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" event={"ID":"8e751995-61e6-44be-a827-81f1e8b25b39","Type":"ContainerDied","Data":"2a9517a54804e0ca505ba1483f713d8e332df2b9074cc821534bf6803be1d227"} Oct 31 23:37:33.575418 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.575390 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:33.602298 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.602268 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\") pod \"8e751995-61e6-44be-a827-81f1e8b25b39\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " Oct 31 23:37:33.602492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.602359 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\") pod \"8e751995-61e6-44be-a827-81f1e8b25b39\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " Oct 31 23:37:33.602492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.602412 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\" (UniqueName: \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\") pod \"8e751995-61e6-44be-a827-81f1e8b25b39\" (UID: \"8e751995-61e6-44be-a827-81f1e8b25b39\") " Oct 31 23:37:33.602780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.602759 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca" pod "8e751995-61e6-44be-a827-81f1e8b25b39" (UID: "8e751995-61e6-44be-a827-81f1e8b25b39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:37:33.604510 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.604480 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca" pod "8e751995-61e6-44be-a827-81f1e8b25b39" (UID: "8e751995-61e6-44be-a827-81f1e8b25b39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:37:33.604606 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.604523 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx" pod "8e751995-61e6-44be-a827-81f1e8b25b39" (UID: "8e751995-61e6-44be-a827-81f1e8b25b39"). InnerVolumeSpecName "kube-api-access-d9wcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:33.703493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.703405 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9wcx\" (UniqueName: \"kubernetes.io/projected/8e751995-61e6-44be-a827-81f1e8b25b39-kube-api-access-d9wcx\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:33.703493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.703438 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:33.703493 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:33.703449 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e751995-61e6-44be-a827-81f1e8b25b39-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:34.444954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:34.444852 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" event={"ID":"8e751995-61e6-44be-a827-81f1e8b25b39","Type":"ContainerDied","Data":"75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83"} Oct 31 23:37:34.444954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:34.444876 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f" Oct 31 23:37:34.444954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:34.444909 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a8bba19a78522822a440e24537f1179870a5e1b4ad267a473719aa7a9e8b83" Oct 31 23:37:41.208431 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:41.208383 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f"] Oct 31 23:37:41.211247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:41.211220 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4619f-9dq5f"] Oct 31 23:37:42.141771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:37:42.141726 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e751995-61e6-44be-a827-81f1e8b25b39" path="/var/lib/kubelet/pods/8e751995-61e6-44be-a827-81f1e8b25b39/volumes" Oct 31 23:37:44.888238 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:37:44.888201 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:38:14.888578 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:38:14.888539 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:38:34.812442 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.812344 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9"] Oct 31 23:38:34.812912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.812890 7074 cpu_manager.go:414] "containerMap: removing container" podUID="8e751995-61e6-44be-a827-81f1e8b25b39" containerName="revoke-keys" Oct 31 23:38:34.812984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.812915 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e751995-61e6-44be-a827-81f1e8b25b39" containerName="revoke-keys" Oct 31 23:38:34.813153 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.813138 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e751995-61e6-44be-a827-81f1e8b25b39" containerName="revoke-keys" Oct 31 23:38:34.817007 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.816966 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.831463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.831432 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9"] Oct 31 23:38:34.933099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933063 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933099 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933103 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933129 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94189916-4895-4e72-bf93-762ed5be1e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933242 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933314 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933272 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933343 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94189916-4895-4e72-bf93-762ed5be1e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933419 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933408 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933483 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933432 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:34.933520 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:34.933493 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6l2h\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-kube-api-access-x6l2h\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.034761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034719 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.034761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034765 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034794 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6l2h\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-kube-api-access-x6l2h\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034836 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034860 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034889 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94189916-4895-4e72-bf93-762ed5be1e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.034920 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035118 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035194 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94189916-4895-4e72-bf93-762ed5be1e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035220 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035377 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035451 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035532 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035455 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.035721 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.035695 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94189916-4895-4e72-bf93-762ed5be1e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.037482 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.037455 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94189916-4895-4e72-bf93-762ed5be1e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.037775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.037753 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94189916-4895-4e72-bf93-762ed5be1e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.048245 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.048215 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.048525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.048506 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6l2h\" (UniqueName: \"kubernetes.io/projected/94189916-4895-4e72-bf93-762ed5be1e7c-kube-api-access-x6l2h\") pod \"maas-default-gateway-openshift-default-6886b96f9-l6mf9\" (UID: \"94189916-4895-4e72-bf93-762ed5be1e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.131217 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.131122 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:35.413321 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.413238 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9"] Oct 31 23:38:35.416551 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:38:35.416522 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94189916_4895_4e72_bf93_762ed5be1e7c.slice/crio-44cddba239be5d984a212131a992fbfb8869d40694a4479888a690c16048d0e2 WatchSource:0}: Error finding container 44cddba239be5d984a212131a992fbfb8869d40694a4479888a690c16048d0e2: Status 404 returned error can't find the container with id 44cddba239be5d984a212131a992fbfb8869d40694a4479888a690c16048d0e2 Oct 31 23:38:35.418798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.418768 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:35.418872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.418834 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:35.418872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.418861 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:35.763324 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.763275 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" event={"ID":"94189916-4895-4e72-bf93-762ed5be1e7c","Type":"ContainerStarted","Data":"6ee5eded9220abdd84a99b0e1b1d56f74974003b0e0708ee1c6e5729d871a8e7"} Oct 31 23:38:35.763516 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.763397 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" event={"ID":"94189916-4895-4e72-bf93-762ed5be1e7c","Type":"ContainerStarted","Data":"44cddba239be5d984a212131a992fbfb8869d40694a4479888a690c16048d0e2"} Oct 31 23:38:35.790262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:35.790195 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podStartSLOduration=1.790175606 podStartE2EDuration="1.790175606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:35.786176667 +0000 UTC m=+2893.768919056" watchObservedRunningTime="2026-10-31 23:38:35.790175606 +0000 UTC m=+2893.772917986" Oct 31 23:38:36.131729 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:36.131641 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:36.133210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:36.133185 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:38:36.133330 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:36.133249 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:38:37.135587 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:37.135555 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:37.775024 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:37.774971 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:37.776055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:37.776034 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:38:44.888164 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:38:44.888124 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:38:49.821018 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.820948 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq"] Oct 31 23:38:49.825407 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.825381 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.841234 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.841202 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq"] Oct 31 23:38:49.883465 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883433 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883488 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def2e60-10a8-4699-9324-60d8ef08a7e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883560 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883634 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883583 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883661 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883678 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883711 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fxd\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-kube-api-access-h6fxd\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883745 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883732 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.883912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.883872 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984600 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fxd\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-kube-api-access-h6fxd\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984638 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984675 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984723 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984764 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def2e60-10a8-4699-9324-60d8ef08a7e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984785 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984802 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984842 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.984888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.984865 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.985323 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.985182 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.985380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.985320 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.985380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.985368 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.985550 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.985526 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.985616 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.985555 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8def2e60-10a8-4699-9324-60d8ef08a7e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.987113 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.987092 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.987317 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.987301 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.997316 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.997296 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:49.997621 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:49.997600 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fxd\" (UniqueName: \"kubernetes.io/projected/8def2e60-10a8-4699-9324-60d8ef08a7e2-kube-api-access-h6fxd\") pod \"maas-default-gateway-openshift-default-6886b96f9-z4xwq\" (UID: \"8def2e60-10a8-4699-9324-60d8ef08a7e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:50.141380 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.141305 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:50.423073 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.423050 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq"] Oct 31 23:38:50.424968 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:38:50.424946 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8def2e60_10a8_4699_9324_60d8ef08a7e2.slice/crio-d60c01028e944f5993073ceeea591fe075e9141712ab37477c733267b4e31978 WatchSource:0}: Error finding container d60c01028e944f5993073ceeea591fe075e9141712ab37477c733267b4e31978: Status 404 returned error can't find the container with id d60c01028e944f5993073ceeea591fe075e9141712ab37477c733267b4e31978 Oct 31 23:38:50.427208 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.427174 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:50.427300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.427239 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:50.427300 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.427272 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:38:50.852789 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.852754 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" event={"ID":"8def2e60-10a8-4699-9324-60d8ef08a7e2","Type":"ContainerStarted","Data":"0767882b1e9aebea7635496e58b920c2a0d0eaf28c2736cdcf55c963f7fb9c5b"} Oct 31 23:38:50.853262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.852806 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" event={"ID":"8def2e60-10a8-4699-9324-60d8ef08a7e2","Type":"ContainerStarted","Data":"d60c01028e944f5993073ceeea591fe075e9141712ab37477c733267b4e31978"} Oct 31 23:38:50.880554 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:50.880507 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podStartSLOduration=1.8804925959999998 podStartE2EDuration="1.880492596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:50.878423313 +0000 UTC m=+2908.861165690" watchObservedRunningTime="2026-10-31 23:38:50.880492596 +0000 UTC m=+2908.863234974" Oct 31 23:38:51.142148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:51.142068 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:51.143780 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:51.143755 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:38:51.143912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:51.143804 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:38:52.146680 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:52.146649 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:52.865302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:52.865272 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:38:52.866312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:38:52.866291 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:39:14.888391 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:39:14.888357 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:39:44.887820 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:39:44.887785 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:39:53.817233 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:53.817177 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:39:53.817707 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:53.817278 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:39:53.907175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:53.907131 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:39:53.907342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:53.907187 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:39:55.242411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.242330 7074 generic.go:356] "Generic (PLEG): container finished" podID="94189916-4895-4e72-bf93-762ed5be1e7c" containerID="6ee5eded9220abdd84a99b0e1b1d56f74974003b0e0708ee1c6e5729d871a8e7" exitCode=0 Oct 31 23:39:55.242897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.242399 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" event={"ID":"94189916-4895-4e72-bf93-762ed5be1e7c","Type":"ContainerDied","Data":"6ee5eded9220abdd84a99b0e1b1d56f74974003b0e0708ee1c6e5729d871a8e7"} Oct 31 23:39:55.242897 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.242813 7074 scope.go:122] "RemoveContainer" containerID="6ee5eded9220abdd84a99b0e1b1d56f74974003b0e0708ee1c6e5729d871a8e7" Oct 31 23:39:55.244202 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244172 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:55.244304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244240 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:55.244304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244245 7074 generic.go:356] "Generic (PLEG): container finished" podID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerID="0767882b1e9aebea7635496e58b920c2a0d0eaf28c2736cdcf55c963f7fb9c5b" exitCode=0 Oct 31 23:39:55.244304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244272 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:55.244420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244317 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" event={"ID":"8def2e60-10a8-4699-9324-60d8ef08a7e2","Type":"ContainerDied","Data":"0767882b1e9aebea7635496e58b920c2a0d0eaf28c2736cdcf55c963f7fb9c5b"} Oct 31 23:39:55.244722 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.244650 7074 scope.go:122] "RemoveContainer" containerID="0767882b1e9aebea7635496e58b920c2a0d0eaf28c2736cdcf55c963f7fb9c5b" Oct 31 23:39:55.245637 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.245611 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:55.245746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.245678 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:55.245746 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:55.245711 7074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 31 23:39:56.295539 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:56.295490 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" event={"ID":"94189916-4895-4e72-bf93-762ed5be1e7c","Type":"ContainerStarted","Data":"a130ced1f458eb1fa84d7a36e29a6a70ff311e6c9ce3f2530976532b05ee5234"} Oct 31 23:39:56.302500 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:56.302469 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" event={"ID":"8def2e60-10a8-4699-9324-60d8ef08a7e2","Type":"ContainerStarted","Data":"785fa3de6dc7c476f1b6f83ac4d2bda6587837bcf85fb951fda982707211e195"} Oct 31 23:39:57.132307 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.132273 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:39:57.132495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.132435 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:39:57.132495 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.132484 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:39:57.141900 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.141877 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:39:57.142033 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.142013 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:39:57.142080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:57.142053 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:39:58.131767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:58.131735 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:39:58.132183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:58.131795 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:39:58.142355 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:58.142321 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:39:58.142506 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:58.142392 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:39:59.132093 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:59.132057 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:39:59.132491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:59.132114 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:39:59.141758 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:59.141737 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:39:59.141837 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:39:59.141782 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:40:00.132203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:00.132165 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:40:00.132671 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:00.132224 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:40:00.141870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:00.141842 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:40:00.142017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:00.141889 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:40:01.131692 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:01.131654 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:40:01.131874 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:01.131721 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:40:01.141944 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:01.141916 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:40:01.142274 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:01.141967 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:40:02.132374 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:02.132336 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l6mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 23:40:02.132567 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:02.132405 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" podUID="94189916-4895-4e72-bf93-762ed5be1e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 23:40:02.141669 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:02.141642 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z4xwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 23:40:02.141800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:02.141682 7074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" podUID="8def2e60-10a8-4699-9324-60d8ef08a7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 23:40:03.588528 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.588438 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:40:03.589617 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.589591 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:40:03.590413 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.590387 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l6mf9" Oct 31 23:40:03.642810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.642781 7074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:40:03.643189 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.643165 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:40:03.643929 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:03.643907 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z4xwq" Oct 31 23:40:14.587146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:14.587095 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:40:14.587693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:14.587180 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:40:14.911811 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:40:14.911690 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:40:22.681726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:22.681694 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:40:22.684903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:40:22.684866 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:40:44.888757 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:40:44.888715 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:41:14.888434 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:41:14.888398 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:41:44.887824 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:41:44.887785 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:42:14.937173 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:42:14.937127 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:42:23.022203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:42:23.022168 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8"] Oct 31 23:42:23.026710 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:42:23.026684 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2ce870-r5kk8"] Oct 31 23:42:24.141703 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:42:24.141672 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe30110f-43c6-4a67-ad18-1ebb01287ba5" path="/var/lib/kubelet/pods/fe30110f-43c6-4a67-ad18-1ebb01287ba5/volumes" Oct 31 23:42:44.932304 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:42:44.932264 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:42:59.596203 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:42:59.596146 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:42:59.597583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:42:59.597534 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:43:14.888365 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:43:14.888284 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:43:23.419277 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:23.419243 7074 scope.go:122] "RemoveContainer" containerID="aaf507a4552898f209e6a9614f3d6f677790ea25d158fc617b39de73adbc76e8" Oct 31 23:43:42.715098 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.715059 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf"] Oct 31 23:43:42.719154 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.719130 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.728023 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.727938 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:43:42.728158 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.728117 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:43:42.730711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.730683 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf"] Oct 31 23:43:42.861346 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.861312 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.861522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.861445 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.861522 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.861509 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.962784 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.962740 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.962982 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.962813 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.962982 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.962875 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.963515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.963483 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.965381 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:42.965324 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:42.969321 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:43:42.969304 7074 projected.go:196] Error preparing data for projected volume kube-api-access-tch5c for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 23:43:42.969415 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:43:42.969368 7074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c podName:f33d16ae-04ec-4687-a7e2-608cfd4f0310 nodeName:}" failed. No retries permitted until 2026-10-31 23:43:43.469354127 +0000 UTC m=+3201.452096498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tch5c" (UniqueName: "kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c") pod "maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" (UID: "f33d16ae-04ec-4687-a7e2-608cfd4f0310") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 23:43:43.570247 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:43.570203 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:43.582836 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:43.582806 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") pod \"maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:43.633662 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:43.633627 7074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" secret="" err="secret \"maas-api-cleanup-dockercfg-67hcr\" not found" Oct 31 23:43:43.633834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:43.633680 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:43.924411 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:43.924297 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf"] Oct 31 23:43:44.681146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:44.681054 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" event={"ID":"f33d16ae-04ec-4687-a7e2-608cfd4f0310","Type":"ContainerStarted","Data":"702c628146953be1b5ca5c7ce900335fa9877747d976385f764025b7ecbfda15"} Oct 31 23:43:44.681146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:44.681134 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" event={"ID":"f33d16ae-04ec-4687-a7e2-608cfd4f0310","Type":"ContainerStarted","Data":"51c179c00f555dc05fa01567b303f2e4282fbc3cc5c6b97fb29caffaec40bf96"} Oct 31 23:43:44.681399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:44.681385 7074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" secret="" err="secret \"maas-api-cleanup-dockercfg-67hcr\" not found" Oct 31 23:43:44.701258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:44.701205 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" podStartSLOduration=2.701192098 podStartE2EDuration="2.701192098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:44.699925592 +0000 UTC m=+3202.682667968" watchObservedRunningTime="2026-10-31 23:43:44.701192098 +0000 UTC m=+3202.683934513" Oct 31 23:43:44.889920 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:43:44.889875 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:43:45.687726 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:45.687691 7074 generic.go:356] "Generic (PLEG): container finished" podID="f33d16ae-04ec-4687-a7e2-608cfd4f0310" containerID="702c628146953be1b5ca5c7ce900335fa9877747d976385f764025b7ecbfda15" exitCode=0 Oct 31 23:43:45.688165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:45.687778 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" event={"ID":"f33d16ae-04ec-4687-a7e2-608cfd4f0310","Type":"ContainerDied","Data":"702c628146953be1b5ca5c7ce900335fa9877747d976385f764025b7ecbfda15"} Oct 31 23:43:46.829636 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:46.829610 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:47.006933 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.006900 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\") pod \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " Oct 31 23:43:47.007132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.006939 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\") pod \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " Oct 31 23:43:47.007132 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.007023 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") pod \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\" (UID: \"f33d16ae-04ec-4687-a7e2-608cfd4f0310\") " Oct 31 23:43:47.007359 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.007329 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca" pod "f33d16ae-04ec-4687-a7e2-608cfd4f0310" (UID: "f33d16ae-04ec-4687-a7e2-608cfd4f0310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:43:47.007609 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.007578 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:47.009176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.009147 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c" pod "f33d16ae-04ec-4687-a7e2-608cfd4f0310" (UID: "f33d16ae-04ec-4687-a7e2-608cfd4f0310"). InnerVolumeSpecName "kube-api-access-tch5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:43:47.009265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.009177 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca" pod "f33d16ae-04ec-4687-a7e2-608cfd4f0310" (UID: "f33d16ae-04ec-4687-a7e2-608cfd4f0310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:43:47.109227 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.109177 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tch5c\" (UniqueName: \"kubernetes.io/projected/f33d16ae-04ec-4687-a7e2-608cfd4f0310-kube-api-access-tch5c\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:47.109409 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.109231 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33d16ae-04ec-4687-a7e2-608cfd4f0310-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:47.700915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.700882 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" Oct 31 23:43:47.700915 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.700904 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf" event={"ID":"f33d16ae-04ec-4687-a7e2-608cfd4f0310","Type":"ContainerDied","Data":"51c179c00f555dc05fa01567b303f2e4282fbc3cc5c6b97fb29caffaec40bf96"} Oct 31 23:43:47.701195 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:47.700951 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c179c00f555dc05fa01567b303f2e4282fbc3cc5c6b97fb29caffaec40bf96" Oct 31 23:43:53.775439 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:53.775373 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf"] Oct 31 23:43:53.780012 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:53.778439 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d08af8f-wzfnf"] Oct 31 23:43:54.141350 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:43:54.141268 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33d16ae-04ec-4687-a7e2-608cfd4f0310" path="/var/lib/kubelet/pods/f33d16ae-04ec-4687-a7e2-608cfd4f0310/volumes" Oct 31 23:44:14.889038 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:44:14.888983 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:44:23.440732 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:44:23.440699 7074 scope.go:122] "RemoveContainer" containerID="2a9517a54804e0ca505ba1483f713d8e332df2b9074cc821534bf6803be1d227" Oct 31 23:44:44.888614 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:44:44.888574 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:45:00.141728 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.141699 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:00.144224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.142176 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f33d16ae-04ec-4687-a7e2-608cfd4f0310" containerName="revoke-keys" Oct 31 23:45:00.144224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.142188 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33d16ae-04ec-4687-a7e2-608cfd4f0310" containerName="revoke-keys" Oct 31 23:45:00.144224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.142263 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33d16ae-04ec-4687-a7e2-608cfd4f0310" containerName="revoke-keys" Oct 31 23:45:00.145415 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.145396 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.153337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.153312 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\"" Oct 31 23:45:00.154337 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.154290 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:00.262814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.262764 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.263043 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.262834 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sm8\" (UniqueName: \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.364325 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.364289 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.364517 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.364351 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sm8\" (UniqueName: \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.364968 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.364942 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.376527 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.376501 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sm8\" (UniqueName: \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\") pod \"maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.459545 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.459472 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:00.748798 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:00.748773 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:00.750888 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:45:00.750858 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e57c76_479f_4b04_9ef0_c35da1b2ad24.slice/crio-13ddf1fa90e67e857d4a73fdc30e6ed4467c2b289b92fc5adf6596eb15bec778 WatchSource:0}: Error finding container 13ddf1fa90e67e857d4a73fdc30e6ed4467c2b289b92fc5adf6596eb15bec778: Status 404 returned error can't find the container with id 13ddf1fa90e67e857d4a73fdc30e6ed4467c2b289b92fc5adf6596eb15bec778 Oct 31 23:45:01.117638 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:01.117601 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerStarted","Data":"104d472b2c3f63debbba969636e0c7e079b317f84be3af41879a1647c658e2e3"} Oct 31 23:45:01.117814 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:01.117656 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerStarted","Data":"13ddf1fa90e67e857d4a73fdc30e6ed4467c2b289b92fc5adf6596eb15bec778"} Oct 31 23:45:01.137263 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:01.137216 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" podStartSLOduration=1.137202986 podStartE2EDuration="1.137202986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:45:01.135482314 +0000 UTC m=+3279.118224702" watchObservedRunningTime="2026-10-31 23:45:01.137202986 +0000 UTC m=+3279.119945362" Oct 31 23:45:02.124183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:02.124144 7074 generic.go:356] "Generic (PLEG): container finished" podID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerID="104d472b2c3f63debbba969636e0c7e079b317f84be3af41879a1647c658e2e3" exitCode=60 Oct 31 23:45:02.124612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:02.124228 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerDied","Data":"104d472b2c3f63debbba969636e0c7e079b317f84be3af41879a1647c658e2e3"} Oct 31 23:45:02.124612 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:02.124429 7074 scope.go:122] "RemoveContainer" containerID="104d472b2c3f63debbba969636e0c7e079b317f84be3af41879a1647c658e2e3" Oct 31 23:45:03.131809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:03.131775 7074 generic.go:356] "Generic (PLEG): container finished" podID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerID="4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10" exitCode=60 Oct 31 23:45:03.132282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:03.131857 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerDied","Data":"4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10"} Oct 31 23:45:03.132282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:03.131911 7074 scope.go:122] "RemoveContainer" containerID="104d472b2c3f63debbba969636e0c7e079b317f84be3af41879a1647c658e2e3" Oct 31 23:45:03.132282 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:03.132108 7074 scope.go:122] "RemoveContainer" containerID="4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10" Oct 31 23:45:03.132413 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:45:03.132393 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m_odh-ai-gateway-infra(03e57c76-479f-4b04-9ef0-c35da1b2ad24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" Oct 31 23:45:04.142399 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:04.142369 7074 scope.go:122] "RemoveContainer" containerID="4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10" Oct 31 23:45:04.142777 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:45:04.142680 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m_odh-ai-gateway-infra(03e57c76-479f-4b04-9ef0-c35da1b2ad24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" Oct 31 23:45:14.890039 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:45:14.889971 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:45:15.135302 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:15.135275 7074 scope.go:122] "RemoveContainer" containerID="4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10" Oct 31 23:45:16.208167 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:16.208091 7074 generic.go:356] "Generic (PLEG): container finished" podID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerID="ac118675aa33e1379707cd89235cc5089776ce19e1f63400b79dfa3d3bc732e9" exitCode=60 Oct 31 23:45:16.208555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:16.208161 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerDied","Data":"ac118675aa33e1379707cd89235cc5089776ce19e1f63400b79dfa3d3bc732e9"} Oct 31 23:45:16.208555 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:16.208225 7074 scope.go:122] "RemoveContainer" containerID="4ee4b6e6cb4e1f3cf85da7ab93f6e0357cdf5d1c247487f6cec0b6b36dd27a10" Oct 31 23:45:16.208661 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:16.208553 7074 scope.go:122] "RemoveContainer" containerID="ac118675aa33e1379707cd89235cc5089776ce19e1f63400b79dfa3d3bc732e9" Oct 31 23:45:16.208863 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:45:16.208827 7074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m_odh-ai-gateway-infra(03e57c76-479f-4b04-9ef0-c35da1b2ad24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" Oct 31 23:45:17.237724 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.237687 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:17.394492 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.394462 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:17.470756 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.470721 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\" (UniqueName: \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\") pod \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " Oct 31 23:45:17.470954 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.470812 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\") pod \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\" (UID: \"03e57c76-479f-4b04-9ef0-c35da1b2ad24\") " Oct 31 23:45:17.471213 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.471186 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca" pod "03e57c76-479f-4b04-9ef0-c35da1b2ad24" (UID: "03e57c76-479f-4b04-9ef0-c35da1b2ad24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:45:17.472907 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.472887 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8" pod "03e57c76-479f-4b04-9ef0-c35da1b2ad24" (UID: "03e57c76-479f-4b04-9ef0-c35da1b2ad24"). InnerVolumeSpecName "kube-api-access-q7sm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:45:17.571854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.571815 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e57c76-479f-4b04-9ef0-c35da1b2ad24-maas-api-service-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:17.571854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:17.571851 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7sm8\" (UniqueName: \"kubernetes.io/projected/03e57c76-479f-4b04-9ef0-c35da1b2ad24-kube-api-access-q7sm8\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:18.224434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:18.224405 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" Oct 31 23:45:18.224434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:18.224420 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m" event={"ID":"03e57c76-479f-4b04-9ef0-c35da1b2ad24","Type":"ContainerDied","Data":"13ddf1fa90e67e857d4a73fdc30e6ed4467c2b289b92fc5adf6596eb15bec778"} Oct 31 23:45:18.224668 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:18.224479 7074 scope.go:122] "RemoveContainer" containerID="ac118675aa33e1379707cd89235cc5089776ce19e1f63400b79dfa3d3bc732e9" Oct 31 23:45:18.249748 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:18.249723 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:18.255097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:18.255065 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-44381a-29891505-slq4m"] Oct 31 23:45:20.140805 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:20.140774 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" path="/var/lib/kubelet/pods/03e57c76-479f-4b04-9ef0-c35da1b2ad24/volumes" Oct 31 23:45:22.720706 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:22.720674 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:45:22.726887 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:22.726866 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hd6q_c9fe7d02-92f2-41fd-9122-abdfdc159006/ovn-acl-logging/0.log" Oct 31 23:45:39.854017 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.853959 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z"] Oct 31 23:45:39.854708 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854686 7074 cpu_manager.go:414] "containerMap: removing container" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.854797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854711 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.854797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854734 7074 cpu_manager.go:414] "containerMap: removing container" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.854797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854741 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.854797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854767 7074 cpu_manager.go:414] "containerMap: removing container" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.854797 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854774 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.855083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854902 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.855083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854917 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.855083 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.854932 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e57c76-479f-4b04-9ef0-c35da1b2ad24" containerName="cleanup" Oct 31 23:45:39.858420 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.858400 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:39.866028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.865886 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:45:39.866028 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.865956 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\"" Oct 31 23:45:39.868960 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:39.868923 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z"] Oct 31 23:45:40.032135 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.032103 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.032335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.032151 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.032335 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.032173 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzf25\" (UniqueName: \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.132980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.132868 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.132980 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.132920 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzf25\" (UniqueName: \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.133246 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.133094 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.133619 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.133597 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.135400 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.135374 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.146643 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.146607 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzf25\" (UniqueName: \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.174379 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.174349 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:40.450695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:40.450670 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z"] Oct 31 23:45:40.453338 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:45:40.453305 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf7c681_07b0_434e_826a_40c5686a0842.slice/crio-f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea WatchSource:0}: Error finding container f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea: Status 404 returned error can't find the container with id f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea Oct 31 23:45:41.355491 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:41.355454 7074 generic.go:356] "Generic (PLEG): container finished" podID="0bf7c681-07b0-434e-826a-40c5686a0842" containerID="6df0a19363bc957f7885c3e4671b381d29654eb1c7644f9cce4954e3ec8dd2ca" exitCode=0 Oct 31 23:45:41.355928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:41.355544 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" event={"ID":"0bf7c681-07b0-434e-826a-40c5686a0842","Type":"ContainerDied","Data":"6df0a19363bc957f7885c3e4671b381d29654eb1c7644f9cce4954e3ec8dd2ca"} Oct 31 23:45:41.355928 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:41.355605 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" event={"ID":"0bf7c681-07b0-434e-826a-40c5686a0842","Type":"ContainerStarted","Data":"f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea"} Oct 31 23:45:42.494325 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.494300 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:42.661779 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.661684 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\") pod \"0bf7c681-07b0-434e-826a-40c5686a0842\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " Oct 31 23:45:42.661958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.661777 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\") pod \"0bf7c681-07b0-434e-826a-40c5686a0842\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " Oct 31 23:45:42.661958 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.661925 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\" (UniqueName: \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\") pod \"0bf7c681-07b0-434e-826a-40c5686a0842\" (UID: \"0bf7c681-07b0-434e-826a-40c5686a0842\") " Oct 31 23:45:42.662230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.662195 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca" pod "0bf7c681-07b0-434e-826a-40c5686a0842" (UID: "0bf7c681-07b0-434e-826a-40c5686a0842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:45:42.662369 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.662353 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:42.663940 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.663916 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca" pod "0bf7c681-07b0-434e-826a-40c5686a0842" (UID: "0bf7c681-07b0-434e-826a-40c5686a0842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:45:42.664146 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.663982 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25" pod "0bf7c681-07b0-434e-826a-40c5686a0842" (UID: "0bf7c681-07b0-434e-826a-40c5686a0842"). InnerVolumeSpecName "kube-api-access-jzf25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:45:42.763129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.763098 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf7c681-07b0-434e-826a-40c5686a0842-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:42.763129 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:42.763125 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzf25\" (UniqueName: \"kubernetes.io/projected/0bf7c681-07b0-434e-826a-40c5686a0842-kube-api-access-jzf25\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:43.367675 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:43.367645 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" Oct 31 23:45:43.367859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:43.367644 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z" event={"ID":"0bf7c681-07b0-434e-826a-40c5686a0842","Type":"ContainerDied","Data":"f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea"} Oct 31 23:45:43.367859 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:43.367773 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3fa14ac3b6a67c6f94e0d9632346320bdcae99505ee3cc3e8aa0072e5b633ea" Oct 31 23:45:44.889108 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:45:44.889061 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:45:55.319922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:55.319881 7074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z"] Oct 31 23:45:55.322775 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:55.322746 7074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44381a-rzm2z"] Oct 31 23:45:56.141891 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:45:56.141861 7074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf7c681-07b0-434e-826a-40c5686a0842" path="/var/lib/kubelet/pods/0bf7c681-07b0-434e-826a-40c5686a0842/volumes" Oct 31 23:46:14.540618 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.540512 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r"] Oct 31 23:46:14.541165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.540981 7074 cpu_manager.go:414] "containerMap: removing container" podUID="0bf7c681-07b0-434e-826a-40c5686a0842" containerName="revoke-keys" Oct 31 23:46:14.541165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.541006 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf7c681-07b0-434e-826a-40c5686a0842" containerName="revoke-keys" Oct 31 23:46:14.541165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.541101 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf7c681-07b0-434e-826a-40c5686a0842" containerName="revoke-keys" Oct 31 23:46:14.544576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.544558 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.552639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.552610 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:46:14.552639 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.552625 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\"" Oct 31 23:46:14.556019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.555664 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r"] Oct 31 23:46:14.637629 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.637594 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.637813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.637656 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.637813 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.637785 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7whv\" (UniqueName: \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.738430 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.738389 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.738619 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.738455 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.738619 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.738529 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7whv\" (UniqueName: \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.739254 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.739230 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.740855 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.740821 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.750525 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.750502 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7whv\" (UniqueName: \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\") pod \"maas-api-revoke-sub-f03629db-mtx4r\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.860119 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:14.860040 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:14.889800 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:46:14.889765 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:46:15.135507 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:15.135481 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r"] Oct 31 23:46:15.137649 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:46:15.137624 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521280b7_a5ee_4489_9418_52ab9031baa8.slice/crio-19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93 WatchSource:0}: Error finding container 19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93: Status 404 returned error can't find the container with id 19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93 Oct 31 23:46:15.574176 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:15.574140 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" event={"ID":"521280b7-a5ee-4489-9418-52ab9031baa8","Type":"ContainerStarted","Data":"e1834efd0c8d597843ef486a825b0537bc73930678d9f808dbfecea703eab0c2"} Oct 31 23:46:15.574607 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:15.574196 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" event={"ID":"521280b7-a5ee-4489-9418-52ab9031baa8","Type":"ContainerStarted","Data":"19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93"} Oct 31 23:46:15.593265 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:15.593217 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" podStartSLOduration=1.5932037970000001 podStartE2EDuration="1.593203797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:46:15.592306049 +0000 UTC m=+3353.575048433" watchObservedRunningTime="2026-10-31 23:46:15.593203797 +0000 UTC m=+3353.575946174" Oct 31 23:46:16.580787 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:16.580751 7074 generic.go:356] "Generic (PLEG): container finished" podID="521280b7-a5ee-4489-9418-52ab9031baa8" containerID="e1834efd0c8d597843ef486a825b0537bc73930678d9f808dbfecea703eab0c2" exitCode=0 Oct 31 23:46:16.581250 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:16.580806 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" event={"ID":"521280b7-a5ee-4489-9418-52ab9031baa8","Type":"ContainerDied","Data":"e1834efd0c8d597843ef486a825b0537bc73930678d9f808dbfecea703eab0c2"} Oct 31 23:46:17.731696 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.731672 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:17.772097 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.772065 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\" (UniqueName: \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\") pod \"521280b7-a5ee-4489-9418-52ab9031baa8\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " Oct 31 23:46:17.772283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.772263 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\") pod \"521280b7-a5ee-4489-9418-52ab9031baa8\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " Oct 31 23:46:17.772354 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.772312 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\") pod \"521280b7-a5ee-4489-9418-52ab9031baa8\" (UID: \"521280b7-a5ee-4489-9418-52ab9031baa8\") " Oct 31 23:46:17.772588 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.772566 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca" pod "521280b7-a5ee-4489-9418-52ab9031baa8" (UID: "521280b7-a5ee-4489-9418-52ab9031baa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:46:17.772690 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.772675 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:17.774116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.774090 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv" pod "521280b7-a5ee-4489-9418-52ab9031baa8" (UID: "521280b7-a5ee-4489-9418-52ab9031baa8"). InnerVolumeSpecName "kube-api-access-s7whv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:46:17.774254 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.774210 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca" pod "521280b7-a5ee-4489-9418-52ab9031baa8" (UID: "521280b7-a5ee-4489-9418-52ab9031baa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:46:17.873740 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.873657 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521280b7-a5ee-4489-9418-52ab9031baa8-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:17.873740 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:17.873687 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7whv\" (UniqueName: \"kubernetes.io/projected/521280b7-a5ee-4489-9418-52ab9031baa8-kube-api-access-s7whv\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:18.594713 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:18.594684 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" Oct 31 23:46:18.594899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:18.594683 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mtx4r" event={"ID":"521280b7-a5ee-4489-9418-52ab9031baa8","Type":"ContainerDied","Data":"19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93"} Oct 31 23:46:18.594899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:18.594806 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f83160ceeec9eaa2869eab00538091fbd646fc83bae7c9d196894d8412ae93" Oct 31 23:46:29.588727 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:29.588150 7074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m522g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:46:29.588727 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:29.588260 7074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m522g" podUID="3bb1470c-1020-463d-a1ef-d00399fb13d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:46:31.010016 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.009950 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh"] Oct 31 23:46:31.010570 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.010555 7074 cpu_manager.go:414] "containerMap: removing container" podUID="521280b7-a5ee-4489-9418-52ab9031baa8" containerName="revoke-keys" Oct 31 23:46:31.010616 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.010570 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521280b7-a5ee-4489-9418-52ab9031baa8" containerName="revoke-keys" Oct 31 23:46:31.010686 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.010664 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="521280b7-a5ee-4489-9418-52ab9031baa8" containerName="revoke-keys" Oct 31 23:46:31.014092 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.014071 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.025589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.022225 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\"" Oct 31 23:46:31.025589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.022314 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:46:31.025589 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.023345 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh"] Oct 31 23:46:31.143215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.143183 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.143428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.143231 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.143428 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.143335 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2n2\" (UniqueName: \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.244148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.244108 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.244340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.244177 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.244340 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.244221 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2n2\" (UniqueName: \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.245039 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.244979 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.246599 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.246577 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.257444 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.257420 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2n2\" (UniqueName: \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\") pod \"maas-api-revoke-sub-0165d7d2-7mjjh\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.332284 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.332193 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:31.607921 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.607890 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh"] Oct 31 23:46:31.610074 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:46:31.610049 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1eaaa28_6895_4d3a_8f56_1f7920346225.slice/crio-bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2 WatchSource:0}: Error finding container bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2: Status 404 returned error can't find the container with id bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2 Oct 31 23:46:31.724281 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:31.724238 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" event={"ID":"f1eaaa28-6895-4d3a-8f56-1f7920346225","Type":"ContainerStarted","Data":"bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2"} Oct 31 23:46:32.731847 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:32.731749 7074 generic.go:356] "Generic (PLEG): container finished" podID="f1eaaa28-6895-4d3a-8f56-1f7920346225" containerID="3a2b98e6f10b9f9154fb193bc6094385000893327ffeb921d8c5415bce2742bd" exitCode=0 Oct 31 23:46:32.732286 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:32.731840 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" event={"ID":"f1eaaa28-6895-4d3a-8f56-1f7920346225","Type":"ContainerDied","Data":"3a2b98e6f10b9f9154fb193bc6094385000893327ffeb921d8c5415bce2742bd"} Oct 31 23:46:33.885019 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.884974 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:33.972576 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.972543 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\" (UniqueName: \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\") pod \"f1eaaa28-6895-4d3a-8f56-1f7920346225\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " Oct 31 23:46:33.972752 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.972645 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\") pod \"f1eaaa28-6895-4d3a-8f56-1f7920346225\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " Oct 31 23:46:33.972803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.972787 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\") pod \"f1eaaa28-6895-4d3a-8f56-1f7920346225\" (UID: \"f1eaaa28-6895-4d3a-8f56-1f7920346225\") " Oct 31 23:46:33.973159 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.973131 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca" pod "f1eaaa28-6895-4d3a-8f56-1f7920346225" (UID: "f1eaaa28-6895-4d3a-8f56-1f7920346225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:46:33.973294 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.973260 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:33.974679 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.974658 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2" pod "f1eaaa28-6895-4d3a-8f56-1f7920346225" (UID: "f1eaaa28-6895-4d3a-8f56-1f7920346225"). InnerVolumeSpecName "kube-api-access-bh2n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:46:33.975044 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:33.975019 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca" pod "f1eaaa28-6895-4d3a-8f56-1f7920346225" (UID: "f1eaaa28-6895-4d3a-8f56-1f7920346225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:46:34.074423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:34.074388 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eaaa28-6895-4d3a-8f56-1f7920346225-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:34.074423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:34.074416 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh2n2\" (UniqueName: \"kubernetes.io/projected/f1eaaa28-6895-4d3a-8f56-1f7920346225-kube-api-access-bh2n2\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:34.744553 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:34.744520 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" Oct 31 23:46:34.744737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:34.744525 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7mjjh" event={"ID":"f1eaaa28-6895-4d3a-8f56-1f7920346225","Type":"ContainerDied","Data":"bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2"} Oct 31 23:46:34.744737 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:34.744645 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf50d1c071bd1edfc5b15b15cfaa48ce0cb5883aac3af01a0a442fb216197fe2" Oct 31 23:46:41.737258 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.737223 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h"] Oct 31 23:46:41.737701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.737682 7074 cpu_manager.go:414] "containerMap: removing container" podUID="f1eaaa28-6895-4d3a-8f56-1f7920346225" containerName="revoke-keys" Oct 31 23:46:41.737701 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.737693 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eaaa28-6895-4d3a-8f56-1f7920346225" containerName="revoke-keys" Oct 31 23:46:41.737810 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.737800 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eaaa28-6895-4d3a-8f56-1f7920346225" containerName="revoke-keys" Oct 31 23:46:41.741122 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.741094 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.749178 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.749154 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9nfr7\"" Oct 31 23:46:41.749338 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.749176 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:46:41.754328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.754301 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h"] Oct 31 23:46:41.861729 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.861681 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48rp\" (UniqueName: \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.861939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.861760 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.861939 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.861916 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.962869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.962828 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48rp\" (UniqueName: \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.962869 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.962876 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.963183 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.962942 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.963630 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.963591 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.965238 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.965210 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:41.975055 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:41.975024 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48rp\" (UniqueName: \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\") pod \"maas-api-revoke-sub-6b19338f-jj97h\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:42.056161 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:42.056098 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:42.333163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:42.333126 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h"] Oct 31 23:46:42.335491 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:46:42.335464 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e8a15a_edaa_4b41_b4f3_aabc343d964c.slice/crio-af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b WatchSource:0}: Error finding container af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b: Status 404 returned error can't find the container with id af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b Oct 31 23:46:42.791550 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:42.791516 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" event={"ID":"b8e8a15a-edaa-4b41-b4f3-aabc343d964c","Type":"ContainerStarted","Data":"051ff5c1952e2749a9ddf4d49efd3ed0b79436334ed2a68008a412bdb582f2d3"} Oct 31 23:46:42.792004 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:42.791576 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" event={"ID":"b8e8a15a-edaa-4b41-b4f3-aabc343d964c","Type":"ContainerStarted","Data":"af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b"} Oct 31 23:46:42.814235 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:42.814185 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" podStartSLOduration=1.814171856 podStartE2EDuration="1.814171856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:46:42.812647692 +0000 UTC m=+3380.795390075" watchObservedRunningTime="2026-10-31 23:46:42.814171856 +0000 UTC m=+3380.796914233" Oct 31 23:46:43.798794 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:43.798761 7074 generic.go:356] "Generic (PLEG): container finished" podID="b8e8a15a-edaa-4b41-b4f3-aabc343d964c" containerID="051ff5c1952e2749a9ddf4d49efd3ed0b79436334ed2a68008a412bdb582f2d3" exitCode=0 Oct 31 23:46:43.799259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:43.798806 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" event={"ID":"b8e8a15a-edaa-4b41-b4f3-aabc343d964c","Type":"ContainerDied","Data":"051ff5c1952e2749a9ddf4d49efd3ed0b79436334ed2a68008a412bdb582f2d3"} Oct 31 23:46:44.901594 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:46:44.901553 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:46:44.943304 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:44.943276 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:46:45.099538 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.099432 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\") pod \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " Oct 31 23:46:45.099766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.099552 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\") pod \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " Oct 31 23:46:45.099766 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.099625 7074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\" (UniqueName: \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\") pod \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\" (UID: \"b8e8a15a-edaa-4b41-b4f3-aabc343d964c\") " Oct 31 23:46:45.099917 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.099891 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca" pod "b8e8a15a-edaa-4b41-b4f3-aabc343d964c" (UID: "b8e8a15a-edaa-4b41-b4f3-aabc343d964c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:46:45.100074 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.100058 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-openshift-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:45.101623 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.101603 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca" pod "b8e8a15a-edaa-4b41-b4f3-aabc343d964c" (UID: "b8e8a15a-edaa-4b41-b4f3-aabc343d964c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:46:45.101719 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.101700 7074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp" pod "b8e8a15a-edaa-4b41-b4f3-aabc343d964c" (UID: "b8e8a15a-edaa-4b41-b4f3-aabc343d964c"). InnerVolumeSpecName "kube-api-access-j48rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:46:45.201139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.201092 7074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-maas-api-xks-ca\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:45.201139 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.201129 7074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48rp\" (UniqueName: \"kubernetes.io/projected/b8e8a15a-edaa-4b41-b4f3-aabc343d964c-kube-api-access-j48rp\") on node \"ip-10-0-131-157.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:46:45.811192 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.811154 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" event={"ID":"b8e8a15a-edaa-4b41-b4f3-aabc343d964c","Type":"ContainerDied","Data":"af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b"} Oct 31 23:46:45.811463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.811205 7074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af371ce886c7f2ce4f58aa2155f939fce260a2ea7f294f7f3a24ff38e3718e8b" Oct 31 23:46:45.811463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:46:45.811171 7074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jj97h" Oct 31 23:47:03.376312 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:03.376281 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d58dbf686-blgqt_6983aade-db19-4509-b5d1-dc7c81f8bb88/manager/0.log" Oct 31 23:47:03.567140 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:03.567105 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xzrsr_ea06ed76-cafd-4c4d-8c0e-b687efd71a0a/manager/0.log" Oct 31 23:47:04.240392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:04.240359 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:47:04.453666 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:04.453638 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fmkf2_fbcbb847-ae4b-43ce-bd43-9ab91203d732/server/0.log" Oct 31 23:47:04.854738 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:04.854702 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qw4q4_b829b94d-8617-461c-9338-114f98be38b9/proxy/0.log" Oct 31 23:47:06.207162 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:06.207128 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7mjjh_f1eaaa28-6895-4d3a-8f56-1f7920346225/revoke-keys/0.log" Oct 31 23:47:06.906273 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:06.906247 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jj97h_b8e8a15a-edaa-4b41-b4f3-aabc343d964c/revoke-keys/0.log" Oct 31 23:47:07.087040 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:07.087013 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mtx4r_521280b7-a5ee-4489-9418-52ab9031baa8/revoke-keys/0.log" Oct 31 23:47:07.268854 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:07.268824 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qj74q_d0f23d37-4482-4829-8b98-93819e55af01/postgres/0.log" Oct 31 23:47:09.036767 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:09.036732 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dcdms_395fc085-9125-4599-bf52-aa25d1f7583a/manager/0.log" Oct 31 23:47:11.397066 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:11.397030 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j82k6_c436d9bc-d717-4c14-8eac-ea035d73abbe/discovery/0.log" Oct 31 23:47:11.580712 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:11.580677 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-bxxtg_f1af8a48-7955-498e-a9da-6030072373b6/kube-auth-proxy/0.log" Oct 31 23:47:12.519679 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:12.519648 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/1.log" Oct 31 23:47:12.896436 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:12.896350 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m522g_3bb1470c-1020-463d-a1ef-d00399fb13d4/istio-proxy/0.log" Oct 31 23:47:13.559423 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:13.559393 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/1.log" Oct 31 23:47:13.936058 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:13.935966 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq_9228d705-aae1-4adc-9fe5-9ce1e18afabc/istio-proxy/0.log" Oct 31 23:47:14.121372 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:14.121339 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xdxrp_8b58f6ad-0951-4522-bb08-2c57d176c2b3/payload-pre-processing/0.log" Oct 31 23:47:14.310392 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:14.310356 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:47:14.887848 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:47:14.887802 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:47:15.411903 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:15.411871 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/0.log" Oct 31 23:47:15.952283 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:15.952251 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/0.log" Oct 31 23:47:16.874695 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:16.874664 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr_de3707bc-9fcd-4ae0-8942-1b95d920d785/main/0.log" Oct 31 23:47:17.227912 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:17.227831 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lqrb6_1586d3a6-8548-4363-9e8c-0b5878e019f0/main/0.log" Oct 31 23:47:17.593739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:17.593702 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg_8aa2b701-0d76-4fb1-9246-d86cd4589868/main/0.log" Oct 31 23:47:18.147421 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:18.147389 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr_937794d1-b7e8-4afe-8915-c4916cd7c9cf/main/0.log" Oct 31 23:47:28.270761 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:28.270722 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d58dbf686-blgqt_6983aade-db19-4509-b5d1-dc7c81f8bb88/manager/0.log" Oct 31 23:47:28.481596 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:28.481553 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xzrsr_ea06ed76-cafd-4c4d-8c0e-b687efd71a0a/manager/0.log" Oct 31 23:47:29.096377 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:29.096346 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:47:29.301211 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:29.301180 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fmkf2_fbcbb847-ae4b-43ce-bd43-9ab91203d732/server/0.log" Oct 31 23:47:29.704387 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:29.704339 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qw4q4_b829b94d-8617-461c-9338-114f98be38b9/proxy/0.log" Oct 31 23:47:30.637558 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:30.637530 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7mjjh_f1eaaa28-6895-4d3a-8f56-1f7920346225/revoke-keys/0.log" Oct 31 23:47:31.339163 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:31.339126 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jj97h_b8e8a15a-edaa-4b41-b4f3-aabc343d964c/revoke-keys/0.log" Oct 31 23:47:31.504179 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:31.504147 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mtx4r_521280b7-a5ee-4489-9418-52ab9031baa8/revoke-keys/0.log" Oct 31 23:47:31.687030 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:31.686893 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qj74q_d0f23d37-4482-4829-8b98-93819e55af01/postgres/0.log" Oct 31 23:47:33.046711 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:33.046678 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dcdms_395fc085-9125-4599-bf52-aa25d1f7583a/manager/0.log" Oct 31 23:47:34.706899 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:34.706864 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j82k6_c436d9bc-d717-4c14-8eac-ea035d73abbe/discovery/0.log" Oct 31 23:47:34.889803 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:34.889769 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-bxxtg_f1af8a48-7955-498e-a9da-6030072373b6/kube-auth-proxy/0.log" Oct 31 23:47:35.816554 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:35.816518 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/1.log" Oct 31 23:47:36.161230 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:36.161150 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m522g_3bb1470c-1020-463d-a1ef-d00399fb13d4/istio-proxy/0.log" Oct 31 23:47:36.692513 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:36.692485 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/1.log" Oct 31 23:47:37.052908 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:37.052871 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq_9228d705-aae1-4adc-9fe5-9ce1e18afabc/istio-proxy/0.log" Oct 31 23:47:37.228210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:37.228173 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xdxrp_8b58f6ad-0951-4522-bb08-2c57d176c2b3/payload-pre-processing/0.log" Oct 31 23:47:37.409148 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:37.409054 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:47:38.101403 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:38.101374 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-kljmr_de3707bc-9fcd-4ae0-8942-1b95d920d785/main/0.log" Oct 31 23:47:38.476565 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:38.476479 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lqrb6_1586d3a6-8548-4363-9e8c-0b5878e019f0/main/0.log" Oct 31 23:47:38.846888 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:38.846849 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rcskg_8aa2b701-0d76-4fb1-9246-d86cd4589868/main/0.log" Oct 31 23:47:39.415434 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:47:39.415404 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lthhr_937794d1-b7e8-4afe-8915-c4916cd7c9cf/main/0.log" Oct 31 23:47:44.888232 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:47:44.888189 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:48:11.337094 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:11.337060 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:48:11.591388 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:11.591290 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:48:12.328800 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:12.328765 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:48:12.515801 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:12.515771 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:48:14.888217 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:48:14.888176 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:48:44.887724 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:48:44.887686 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:48:52.884556 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:52.884519 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:48:53.154351 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:53.154250 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69b96f8484-9965q_0442774d-6d62-408e-b222-8f5e57490583/manager/0.log" Oct 31 23:48:53.852127 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:53.852083 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:48:54.024138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:54.024102 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:48:57.627262 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:57.627233 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qqjd2_56499b86-c2fe-429c-88b8-cc5be0e63840/global-pull-secret-syncer/0.log" Oct 31 23:48:58.012937 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:58.012905 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zsmmj_59b2c63f-8550-4152-9759-d522b6b4e520/konnectivity-agent/0.log" Oct 31 23:48:58.042570 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:48:58.042537 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-157.us-east-2.compute.internal_72de75e4a93b82a0133be1d6ed9e0a51/haproxy/0.log" Oct 31 23:49:04.920870 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:04.920781 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dcdms_395fc085-9125-4599-bf52-aa25d1f7583a/manager/0.log" Oct 31 23:49:06.443969 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.443927 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/alertmanager/0.log" Oct 31 23:49:06.472331 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.472302 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/config-reloader/0.log" Oct 31 23:49:06.500475 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.500443 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/kube-rbac-proxy-web/0.log" Oct 31 23:49:06.528704 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.528674 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/kube-rbac-proxy/0.log" Oct 31 23:49:06.558768 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.558738 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/kube-rbac-proxy-metric/0.log" Oct 31 23:49:06.586592 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.586564 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/prom-label-proxy/0.log" Oct 31 23:49:06.614670 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.614643 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dd5ca891-558a-46f3-9693-b4a28951d091/init-config-reloader/0.log" Oct 31 23:49:06.700894 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.700786 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mzdtc_b2252e42-a7dd-4822-83ca-65aff7bb802a/kube-state-metrics/0.log" Oct 31 23:49:06.726080 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.726055 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mzdtc_b2252e42-a7dd-4822-83ca-65aff7bb802a/kube-rbac-proxy-main/0.log" Oct 31 23:49:06.754323 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.754292 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mzdtc_b2252e42-a7dd-4822-83ca-65aff7bb802a/kube-rbac-proxy-self/0.log" Oct 31 23:49:06.786210 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.786182 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d849c4dfc-tbtqx_cc9ed1c1-75ac-42cc-8613-67993c17b9a5/metrics-server/0.log" Oct 31 23:49:06.817175 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.817146 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-gfw6p_27c3dce2-82b9-4868-8ec9-2d480ff15a1c/monitoring-plugin/0.log" Oct 31 23:49:06.854946 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.854921 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wpwb_0d187ad0-d0cd-4442-a0b5-ab7e324cacda/node-exporter/0.log" Oct 31 23:49:06.888193 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.888156 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wpwb_0d187ad0-d0cd-4442-a0b5-ab7e324cacda/kube-rbac-proxy/0.log" Oct 31 23:49:06.916229 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:06.916203 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wpwb_0d187ad0-d0cd-4442-a0b5-ab7e324cacda/init-textfile/0.log" Oct 31 23:49:07.140427 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.140389 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rvwxq_2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/kube-rbac-proxy-main/0.log" Oct 31 23:49:07.168583 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.168554 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rvwxq_2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/kube-rbac-proxy-self/0.log" Oct 31 23:49:07.197367 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.197339 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rvwxq_2c47d3f0-fedf-456a-b4db-dc5115e9c5d6/openshift-state-metrics/0.log" Oct 31 23:49:07.241585 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.241559 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/prometheus/0.log" Oct 31 23:49:07.269454 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.269428 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/config-reloader/0.log" Oct 31 23:49:07.300015 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.299970 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/thanos-sidecar/0.log" Oct 31 23:49:07.333378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.333349 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/kube-rbac-proxy-web/0.log" Oct 31 23:49:07.365317 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.365291 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/kube-rbac-proxy/0.log" Oct 31 23:49:07.376077 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.376048 7074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q"] Oct 31 23:49:07.376771 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.376753 7074 cpu_manager.go:414] "containerMap: removing container" podUID="b8e8a15a-edaa-4b41-b4f3-aabc343d964c" containerName="revoke-keys" Oct 31 23:49:07.376834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.376772 7074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e8a15a-edaa-4b41-b4f3-aabc343d964c" containerName="revoke-keys" Oct 31 23:49:07.376891 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.376882 7074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e8a15a-edaa-4b41-b4f3-aabc343d964c" containerName="revoke-keys" Oct 31 23:49:07.380378 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.380356 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.388919 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.388889 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"kube-root-ca.crt\"" Oct 31 23:49:07.389061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.388962 7074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nmqzm\"/\"default-dockercfg-sstqk\"" Oct 31 23:49:07.389061 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.389056 7074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"openshift-service-ca.crt\"" Oct 31 23:49:07.390215 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.390192 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q"] Oct 31 23:49:07.407171 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.407097 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/kube-rbac-proxy-thanos/0.log" Oct 31 23:49:07.436463 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.436434 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8324d2f9-f4bd-451a-925b-b6e9657b46d2/init-config-reloader/0.log" Oct 31 23:49:07.526851 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.526809 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-lib-modules\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.527305 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.526928 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csgn\" (UniqueName: \"kubernetes.io/projected/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-kube-api-access-9csgn\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.527305 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.526956 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-sys\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.527305 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.527086 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-proc\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.527305 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.527131 7074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-podres\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.569515 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.569485 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc4c664b9-xj859_bc184cbe-a1dd-439c-a678-c3fe086ef90d/telemeter-client/0.log" Oct 31 23:49:07.599861 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.599830 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc4c664b9-xj859_bc184cbe-a1dd-439c-a678-c3fe086ef90d/reload/0.log" Oct 31 23:49:07.627818 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.627786 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-proc\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.627842 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-podres\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.627920 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-proc\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.627961 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-lib-modules\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628020 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-podres\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628068 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628067 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-lib-modules\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628075 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csgn\" (UniqueName: \"kubernetes.io/projected/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-kube-api-access-9csgn\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628097 7074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-sys\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.628322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628171 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc4c664b9-xj859_bc184cbe-a1dd-439c-a678-c3fe086ef90d/kube-rbac-proxy/0.log" Oct 31 23:49:07.628322 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.628175 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-sys\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.640270 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.640242 7074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csgn\" (UniqueName: \"kubernetes.io/projected/24319c8d-0556-4ef9-9b4e-c4735b7a8b37-kube-api-access-9csgn\") pod \"perf-node-gather-daemonset-khl8q\" (UID: \"24319c8d-0556-4ef9-9b4e-c4735b7a8b37\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.663259 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.663188 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/thanos-query/0.log" Oct 31 23:49:07.691328 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.691296 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/kube-rbac-proxy-web/0.log" Oct 31 23:49:07.696342 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.696321 7074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:07.720075 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.720051 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/kube-rbac-proxy/0.log" Oct 31 23:49:07.749224 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.749194 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/prom-label-proxy/0.log" Oct 31 23:49:07.780212 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.780183 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/kube-rbac-proxy-rules/0.log" Oct 31 23:49:07.810508 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.810484 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5896fc6d85-lpp69_94dc71e0-8a72-4a66-a4ab-cb5c3e462d4f/kube-rbac-proxy-metrics/0.log" Oct 31 23:49:07.971950 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:07.971927 7074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q"] Oct 31 23:49:07.974368 ip-10-0-131-157 kubenswrapper[7074]: W1031 23:49:07.974333 7074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod24319c8d_0556_4ef9_9b4e_c4735b7a8b37.slice/crio-1df8deed1ca0b3f9a473a983eae1c65c81256f78cc5776dbc06ff4b14427a05a WatchSource:0}: Error finding container 1df8deed1ca0b3f9a473a983eae1c65c81256f78cc5776dbc06ff4b14427a05a: Status 404 returned error can't find the container with id 1df8deed1ca0b3f9a473a983eae1c65c81256f78cc5776dbc06ff4b14427a05a Oct 31 23:49:08.622389 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:08.622353 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" event={"ID":"24319c8d-0556-4ef9-9b4e-c4735b7a8b37","Type":"ContainerStarted","Data":"221ad84d69684adb6704b4389c2d13c53df34e758de4edea634a11bf946a37aa"} Oct 31 23:49:08.622834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:08.622407 7074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" event={"ID":"24319c8d-0556-4ef9-9b4e-c4735b7a8b37","Type":"ContainerStarted","Data":"1df8deed1ca0b3f9a473a983eae1c65c81256f78cc5776dbc06ff4b14427a05a"} Oct 31 23:49:08.622834 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:08.622427 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:08.641748 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:08.641705 7074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" podStartSLOduration=1.6416932549999999 podStartE2EDuration="1.641693255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 23:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:49:08.640674401 +0000 UTC m=+3526.623416793" watchObservedRunningTime="2026-10-31 23:49:08.641693255 +0000 UTC m=+3526.624435626" Oct 31 23:49:12.066961 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:12.066931 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qbfrp_d9004f5b-8f45-4dfe-b883-d485d3b43582/dns/0.log" Oct 31 23:49:12.095453 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:12.095430 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qbfrp_d9004f5b-8f45-4dfe-b883-d485d3b43582/kube-rbac-proxy/0.log" Oct 31 23:49:12.253963 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:12.253936 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wzqjr_25ef2f42-14fa-4d62-be14-ef9fe9fd92f5/dns-node-resolver/0.log" Oct 31 23:49:12.893116 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:12.893072 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44szm_4544eebe-acaf-4a13-bd4e-c3df8a8fbc0d/node-ca/0.log" Oct 31 23:49:14.639320 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:14.639287 7074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-khl8q" Oct 31 23:49:14.890750 ip-10-0-131-157 kubenswrapper[7074]: E1031 23:49:14.890658 7074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2gk2" containerName="extractor" Oct 31 23:49:15.787046 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:15.787018 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j82k6_c436d9bc-d717-4c14-8eac-ea035d73abbe/discovery/0.log" Oct 31 23:49:15.816138 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:15.816108 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-bxxtg_f1af8a48-7955-498e-a9da-6030072373b6/kube-auth-proxy/0.log" Oct 31 23:49:15.995165 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:15.995122 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/1.log" Oct 31 23:49:15.995356 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:15.995333 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/0.log" Oct 31 23:49:16.053135 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.053054 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m522g_3bb1470c-1020-463d-a1ef-d00399fb13d4/istio-proxy/0.log" Oct 31 23:49:16.171289 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.171252 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/1.log" Oct 31 23:49:16.171468 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.171373 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/0.log" Oct 31 23:49:16.249809 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.249778 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq_9228d705-aae1-4adc-9fe5-9ce1e18afabc/istio-proxy/0.log" Oct 31 23:49:16.282693 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.282664 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xdxrp_8b58f6ad-0951-4522-bb08-2c57d176c2b3/payload-pre-processing/0.log" Oct 31 23:49:16.324725 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:16.324644 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:49:18.597343 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:18.597306 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t48z9_26fd3b3c-9c2e-43b7-9660-ec1ff811348e/serve-healthcheck-canary/0.log" Oct 31 23:49:19.285219 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.285187 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2gk2_eb149490-9112-463a-b5ef-107f84f04d0c/kube-rbac-proxy/0.log" Oct 31 23:49:19.315664 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.315593 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2gk2_eb149490-9112-463a-b5ef-107f84f04d0c/exporter/0.log" Oct 31 23:49:19.347156 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.347126 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2gk2_eb149490-9112-463a-b5ef-107f84f04d0c/extractor/0.log" Oct 31 23:49:19.720635 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.720543 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j82k6_c436d9bc-d717-4c14-8eac-ea035d73abbe/discovery/0.log" Oct 31 23:49:19.732226 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.732197 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-689d5458bb-bxxtg_f1af8a48-7955-498e-a9da-6030072373b6/kube-auth-proxy/0.log" Oct 31 23:49:19.822747 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.822717 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/1.log" Oct 31 23:49:19.822922 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.822904 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l6mf9_94189916-4895-4e72-bf93-762ed5be1e7c/istio-proxy/0.log" Oct 31 23:49:19.856872 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.856847 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m522g_3bb1470c-1020-463d-a1ef-d00399fb13d4/istio-proxy/0.log" Oct 31 23:49:19.914041 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.914013 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/1.log" Oct 31 23:49:19.914188 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.914075 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z4xwq_8def2e60-10a8-4699-9324-60d8ef08a7e2/istio-proxy/0.log" Oct 31 23:49:19.947749 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.947726 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-jtwwq_9228d705-aae1-4adc-9fe5-9ce1e18afabc/istio-proxy/0.log" Oct 31 23:49:19.966353 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.966321 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xdxrp_8b58f6ad-0951-4522-bb08-2c57d176c2b3/payload-pre-processing/0.log" Oct 31 23:49:19.983984 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:19.983910 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-7ql7n_656d49b3-6eff-4b88-8719-08e5fdbfd7d5/payload-processing/0.log" Oct 31 23:49:22.142811 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:22.142783 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-wkct8_38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9/manager/0.log" Oct 31 23:49:22.169739 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:22.169715 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-wkct8_38cdaf26-d9f6-44e5-a1f2-4f0b87507eb9/copy-manifests/0.log" Oct 31 23:49:23.400281 ip-10-0-131-157 kubenswrapper[7074]: I1031 23:49:23.400246 7074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-6pj9v_030efd79-76e3-42bd-a161-e154dca00b6b/jobset-operator/0.log"